A video call method and apparatus

Abstract

The application discloses a video call method and device, which supports real-time drawing and marking on video screen content in a two-party video call or a multi-party video conference, other participants of the video call can watch the content drawn by the drawer in real time, and the effect of 'watching while drawing' is achieved, which helps to improve the quality of information transmission. The terminal device of the sender of the call collects drawing point data and multimedia data in real time, and collects real-time encapsulation data packets and sends them to the receiver through a media server. The data packets encapsulate time stamps and / or serial numbers, and the receiver can synchronize the drawing point data and the multimedia data according to the time stamps and / or the serial numbers, so that the receiver can watch the drawing pattern and hear the voice of the sender user in real time, and the quality and efficiency of information transmission are improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a video call method and apparatus. Background Technology

[0002] Currently, video calling is an effective way to communicate and interact. Besides providing audio and video content to each participant, video calling technology can also offer whiteboard drawing functionality. However, currently, other participants in the video call cannot see the drawing in real time as the artist begins to write or during the drawing process. The drawing is only visible to other participants after the artist stops writing. This means that the artist and other participants cannot see the drawing simultaneously, which prevents other participants from understanding the artist's intended message in a timely manner, reducing the quality and efficiency of communication. Summary of the Invention

[0003] This application provides a video call method and apparatus to improve the quality and efficiency of communication and interaction.

[0004] In a first aspect, embodiments of this application provide a video call method, comprising: a media server receiving a first data stream from a first terminal device, wherein data packets in the first data stream include drawing point data of a user-drawn pattern collected by the first terminal device; the media server receiving a second data stream from the first terminal device, wherein data packets in the second data stream include user audio data and / or user video data of the first terminal device; the media server synchronizing the first data stream and the second data stream according to time parameters of data packets in the first data stream and time parameters of data packets in the second data stream to obtain a synchronization processing result; when data packets in the second data stream include user audio data, the media server rendering the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user-drawn pattern, and sending the user-drawn pattern and the second data stream to the second terminal device; or, when data packets in the second data stream include user video data, the second terminal device rendering the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user-drawn pattern, and rendering the user video data in the data packets of the second data stream to obtain a user video image, superimposing the user-drawn pattern on the user video image to obtain a third video stream, and sending the third video stream to the second terminal device.

[0005] In the above scheme, the media server receives drawing point data, user audio data, and user video data from the terminal device in real time. It then synchronizes these data and renders the user video and drawing point data, eliminating the need for the terminal device to perform rendering and reducing hardware requirements. Furthermore, the terminal device sends drawing point data, audio data, and video data to the media server in real time. After synchronization processing, the media server sends these data to a second terminal device, enabling the second terminal device to display the drawing point data, video data, and play audio data synchronously in real time. This ensures that the user on the second terminal device sees and hears the same audio, improving the quality and efficiency of user interaction.

[0006] In one possible design, the media server renders the drawing points of the user-drawn pattern at the same rate as the first terminal device collects the drawing points of the user-drawn pattern. This scheme further synchronizes the sound heard by the user with the drawn pattern seen.

[0007] In one possible design, when the sampling frequency of the first data stream differs from that of the second data stream, the media server synchronizes the first data stream and the second data stream based on the time parameters of the data packets in the first data stream and the time parameters of the data packets in the second data stream, including:

[0008] When the time parameter includes a timestamp, the media server synchronizes the data packets of the first data stream and the second data stream based on the timestamps and first timestamp increments of the data packets of the first data stream, the timestamps and second timestamp increments of the data packets of the second data stream, wherein the first timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the first data stream, and the second timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the second data stream; or,

[0009] When the time parameter includes a sequence number, the media server synchronizes the data packets of the first data stream and the second data stream based on the sequence number of the data packets of the first data stream, the sequence number of the data packets of the second data stream, and the sequence number increment, wherein the sequence number increment is determined based on the sampling frequency of the first data stream and the sampling frequency of the second data stream.

[0010] The above design provides two feasible synchronization methods.

[0011] In one possible design, the method further includes: the media server storing received data packets belonging to a first data stream in a receive buffer; rendering the drawing point data in the data packets of the first data stream to obtain a user drawing pattern, including: the media server retrieving a first data packet belonging to the first data stream from the receive buffer, and rendering the drawing point data of the first data packet as a first drawing point of the user drawing pattern; when no second data packet belonging to the first data stream is retrieved from the receive buffer at the rate at which the media server renders the drawing points of the user drawing pattern, predicting the drawing point data of the second data packet according to the first N data packets that have already been rendered; wherein, the sequence number of the second data packet is adjacent to the first data packet and the sequence number of the second data packet is greater than the first data packet or the time interval between the timestamp of the second data packet and the timestamp of the first data packet is a time threshold, the time threshold being determined according to the sampling frequency of the first data stream and the sampling frequency of the second data stream; and rendering the predicted drawing point data of the second data packet as a second drawing point of the user drawing pattern.

[0012] By using data prediction, missing drawing point data is predicted, and then the predicted drawing point data is used to replace the missing drawing point data when drawing the pattern, thus preventing deviations in the drawing results.

[0013] In one possible design, the predicted drawing point data of the second data packet is rendered as the second drawing point of the user drawing pattern. The method further includes: when the second data packet is received, replacing the second drawing point with the drawing point rendered according to the drawing point data of the received second data packet.

[0014] In the above design, delayed drawing point data is predicted using data prediction to ensure that the rate of rendering drawing point data is the same as the rate of collecting drawing point data. Therefore, when drawing a pattern, the predicted drawing point data is rendered first, and when delayed drawing point data is received subsequently, any deviations in the predicted drawing point data can be corrected.

[0015] In one possible design, the first data stream is a Real-time Streaming Protocol (RTP) stream, and the second data stream is an RTP stream; or, the first data stream is a Real-time Streaming Control Protocol (RTCP) stream, and the second data stream is an RTCP stream.

[0016] Secondly, embodiments of this application provide a video call method, comprising: a second terminal device receiving a first data stream from a media server, wherein data packets in the first data stream include drawing point data of a user-drawn pattern collected by the first terminal device; the second terminal device receiving a second data stream from the media server, wherein data packets in the second data stream include user audio data and / or user video data of the first terminal device; the second terminal device synchronizing the first data stream and the second data stream according to time parameters of data packets in the first data stream and time parameters of data packets in the second data stream to obtain a synchronization processing result; when data packets in the second data stream include user audio data, the second terminal device rendering data packets in the first data stream according to the synchronization processing result to obtain a user-drawn pattern and displaying the user-drawn pattern, and playing the user audio; or, when data packets in the second data stream include user video data, the second terminal device rendering drawing point data in data packets in the first data stream according to the synchronization processing result to obtain a user-drawn pattern, and rendering user video data in data packets in the second data stream to obtain a user video image, and superimposing the user-drawn pattern on the user video image for display.

[0017] In the above scheme, the second terminal device receives drawing point data, user audio data, and user video data from the first terminal device in real time, and then synchronizes the drawing point data, audio data, and video data, so that the second terminal device can display the drawing point data, video data, and play audio data in real time. This ensures that the user's view and sound are synchronized, thereby improving the quality of user interaction and increasing communication efficiency.

[0018] In one possible design, the rate at which the second terminal device renders the drawing points of the user-drawn pattern is the same as the rate at which the first terminal device acquires the drawing points of the user-drawn pattern.

[0019] In one possible design, when the sampling frequency of the first data stream differs from that of the second data stream, the second terminal device synchronizes the first data stream and the second data stream based on the time parameters of the data packets in the first data stream and the time parameters of the data packets in the second data stream, including:

[0020] When the time parameter includes a timestamp, the second terminal device synchronizes the data packets in the first data stream and the data packets in the second data stream based on the timestamps and first timestamp increments of the data packets in the first data stream, the timestamps and second timestamp increments of the data packets in the second data stream, wherein the first timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the first data stream, and the second timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the second data stream; or,

[0021] When the time parameter includes a sequence number, the second terminal device synchronizes the data packets in the first data stream and the data packets in the second data stream according to the sequence number of the data packets in the first data stream, the sequence number of the data packets in the second data stream, and the sequence number increment. The sequence number increment is determined based on the sampling frequency of the first data stream and the sampling frequency of the second data stream.

[0022] In one possible design, the method further includes:

[0023] The second terminal device stores the received data packets belonging to the first data stream in the receive buffer;

[0024] The second terminal device renders the drawing point data in the data packets of the first data stream to obtain the user-drawn pattern, including:

[0025] The second terminal device obtains a first data packet belonging to the first data stream from the receiving buffer and renders the first data packet as a first drawing point of the user-drawn pattern; when the second terminal device fails to obtain a second data packet belonging to the first data stream from the receiving buffer at the rate at which it renders the drawing points of the user-drawn pattern, it predicts the drawing point data of the second data packet based on the first N data packets belonging to the first data stream that have already been rendered.

[0026] Wherein, the sequence number of the second data packet is adjacent to that of the first data packet and the sequence number of the second data packet is greater than that of the first data packet or the time interval between the timestamp of the second data packet and the timestamp of the first data packet is a time threshold, wherein the time threshold is determined according to the sampling frequency of the first data stream and the sampling frequency of the second data stream;

[0027] The predicted drawing point data of the second data packet is rendered as the second drawing point of the user-drawn pattern.

[0028] In one possible design, the predicted drawing point data of the second data packet is rendered as the second drawing points of the user-drawn pattern, and the method further includes:

[0029] When the second data packet is received, the second drawing point will be replaced by a drawing point rendered based on the drawing point data of the received second data packet.

[0030] In one possible design, the first data stream is a Real-time Streaming Protocol (RTP) stream, and the second data stream is an RTP stream; or, the first data stream is a Real-time Streaming Control Protocol (RTCP) stream, and the second data stream is an RTCP stream.

[0031] Thirdly, embodiments of this application provide a video call method, comprising: a first terminal device responding to a user-drawn pattern drawing operation of the first terminal device, real-time acquisition of drawing points of the user-drawn pattern to obtain drawing point data; the first terminal device real-time acquisition of multimedia data, the multimedia data including user audio data or user video data of the first terminal device; the first terminal device encapsulating the real-time acquisition of the user-drawn pattern drawing point data into a data packet belonging to a first data stream, and encapsulating the real-time acquisition of the multimedia data into a data packet belonging to a second data stream, the data packet of the first data stream including the user-drawn pattern drawing point data, and the data packet of the second data stream including user audio data or user video data of the first terminal device; the data packet in the first data stream carrying a first timestamp, the data packet in the second data stream carrying a second timestamp, the first timestamp and the second timestamp being determined based on the sampling frequency of the drawing points of the user-drawn pattern and the sampling frequency of the user video data or user audio data encapsulated in the second data packet; the first terminal device sending the first data stream and the second data stream to a media server through a session boundary controller (SBC).

[0032] Through the above scheme, the first terminal device of the sender collects drawing point data and multimedia data in real time, and encapsulates data packets while collecting data. The data packets encapsulate timestamps for synchronization. Then, the second terminal device of the receiver can synchronously generate drawing point data and multimedia data according to the timestamps, so that the displayed user-drawn pattern and the heard audio are synchronized.

[0033] In one possible implementation, the method further includes: the first terminal device rendering the drawing point data to obtain a user-drawn pattern and displaying the user-drawn pattern.

[0034] In one possible implementation, the method further includes: the first terminal device receiving a first user-drawn pattern from the media server via the SBC, the first user-drawn pattern being obtained by the media server rendering drawing point data in the data packets of the first data stream; and the first terminal device displaying the first user-drawn pattern.

[0035] In one possible implementation, before the first terminal device receives the first user-drawn pattern sent by the media server through the first SBC, the method further includes: the first terminal device rendering the drawing point data to obtain a second user-drawn pattern and displaying the second user-drawn pattern; after the first terminal device receives the first user-drawn pattern from the media server through the SBC, it displays the first user-drawn pattern, wherein the displayed first user-drawn pattern covers the second user-drawn pattern.

[0036] As an example, the color and / or style of the pattern drawn by the second user may differ from the color and / or style of the pattern drawn by the first user. The user of the first terminal device can determine the user patterns displayed by each video call party based on the displayed user patterns. Furthermore, the user of the first terminal device can adjust the speed at which the content of the user patterns is described, enabling each video call party to accurately understand the content described by the user of the first terminal device, thus improving the user experience.

[0037] In one possible design, the first data stream is a Real-time Streaming Protocol (RTP) stream, and the second data stream is an RTP stream; or, the first data stream is a Real-time Streaming Control Protocol (RTCP) stream, and the second data stream is an RTCP stream.

[0038] Fourthly, this application also provides a communication device applied to a media server. The beneficial effects are described in the first aspect and will not be repeated here. This device has the function of implementing the behavior described in the method example of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. In one possible design, the device structure may include a communication unit and a processing unit, which can perform the corresponding functions in the method example of the first aspect. For details, please refer to the detailed description in the method example, which will not be repeated here.

[0039] Fifthly, embodiments of this application also provide a communication device applied to the terminal device. The beneficial effects are described in the second or third aspects and will not be repeated here. This device has the function of implementing the behaviors described in the method examples of the second or third aspects. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the device structure may include a display unit, a communication unit, and a processing unit, and may also include a playback unit and a acquisition unit. These units can perform the corresponding functions in the method examples of the second or third aspects, as detailed in the method examples, and will not be repeated here.

[0040] Sixthly, embodiments of this application also provide a communication device applied to the media server. The beneficial effects are described in the first aspect and will not be repeated here. The communication device includes a processor and a communication interface, and may also include a memory. The processor is configured to support the media server in performing the corresponding functions of the method in the first aspect via the communication interface. The memory is coupled to the processor and stores necessary program instructions and data for the communication device.

[0041] Seventhly, embodiments of this application also provide a communication device applied to the terminal device. The beneficial effects are described in the second or third aspects and will not be repeated here. The communication device includes a processor and a communication interface, and may also include a memory. The processor is configured to support the media server in performing the corresponding functions of the methods in the second or third aspects described above through the communication interface. The memory is coupled to the processor and stores necessary program instructions and data for the communication device.

[0042] Eighthly, this application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in any of the preceding aspects.

[0043] Ninthly, this application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in any of the preceding aspects.

[0044] In a tenth aspect, this application also provides a computer chip connected to a memory, the chip being used to read and execute a software program stored in the memory, and to perform the method described in any of the preceding aspects. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of a communication system architecture in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of another possible communication system architecture in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the hardware structure of the terminal device in the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of a possible video call process in an embodiment of this application;

[0049] Figure 5A This is a schematic diagram of a possible call interface of the terminal device in the embodiments of this application;

[0050] Figure 5B This is a schematic diagram of another possible call interface of the terminal device in the embodiments of this application;

[0051] Figure 5C This is a schematic diagram of a possible display interface of the terminal device in the embodiments of this application;

[0052] Figure 5D This is a schematic diagram of another possible display interface of the terminal device in the embodiments of this application;

[0053] Figure 6 This is a schematic diagram of the timestamps and sequence numbers of the drawing point data packets and audio data packets encapsulated in the embodiments of this application;

[0054] Figure 7 This is a schematic diagram illustrating the timestamps and sequence numbers for the synchronization of point data packets and audio data packets in an embodiment of this application;

[0055] Figure 8 This is a schematic diagram illustrating the display of missing drawing points and predicted drawing points in a user-drawn pattern in an embodiment of this application.

[0056] Figure 9 This is a schematic diagram of another possible video call process in an embodiment of this application;

[0057] Figure 10 This is a schematic diagram showing the rendering discontinuity caused by time delay in an embodiment of this application.

[0058] Figure 11 This is a schematic diagram of the structure of the communication device 1100 in the embodiments of this application;

[0059] Figure 12 This is a schematic diagram of the structure of the communication device 1200 in the embodiments of this application;

[0060] Figure 13 This is a schematic diagram of the structure of the communication device 1300 in the embodiments of this application;

[0061] Figure 14 This is a schematic diagram of the structure of the communication device 1400 in the embodiments of this application;

[0062] Figure 15 This is a schematic diagram of the structure of the communication device 1500 in the embodiments of this application. Detailed Implementation

[0063] In video call scenarios, to improve communication quality and efficiency, participants can draw content or add annotations. However, currently, other participants cannot see the drawing in real time as the artist begins drawing; the result is only visible after the artist starts drawing. This means the artist and other participants cannot see the drawing simultaneously. Consequently, in remote guidance or collaboration scenarios, other participants cannot clearly understand the content the artist intends to convey or the intended audience, resulting in low quality and efficiency of communication and interaction.

[0064] Based on this, embodiments of this application provide a video call method and apparatus that support real-time drawing and annotation on video content in scenarios of two-party video calls or multi-party video conferences. Other participants in the video call can view the content drawn by the annotator in real time, achieving a "see and draw" effect, which helps improve the quality of information transmission. This increases the information transmission channels in video calls or video conferences, facilitating remote guidance or remote collaboration between two or more parties.

[0065] Video calls can be made, but are not limited to, using Voice over Long-Term Evolution (VoLTE) or social applications (APPs), and can also be used for Voice over New Radio (VoNR) video calls. The following description will use VoLTE as an example for video calls. The embodiments of this application are applicable to two-party video call scenarios, and also to multi-party video call or multi-party video conferencing scenarios.

[0066] See Figure 1 The diagram illustrates a communication system architecture according to an embodiment of this application. The communication system includes one or more session border controllers (SBCs) and one or more media servers. The communication system can enable video calls and / or audio calls.

[0067] The Session Boundary Controller (SBC) can be called a boundary gateway. The media server can also be called an AR media server or an AR media enabler (AME). Two or more terminal devices can conduct voice and video calls through a communication system. During these calls, the AR media enabler processes the media streams generated. The media stream can include user video generated during the video call, as well as audio. User video can include video captured by the user's terminal device's camera (front or rear camera), files opened by the user, or the user's image. For example, the AR media enabler, with its powerful image processing and data computation capabilities, can use AR technology to perform logical operations, image rendering, and virtual scene synthesis on the received media stream. The AR media server can be deployed as a container service. It can also be implemented using a single or virtual machine. Furthermore, the AR media server can include a single or multiple processors, or be implemented using one or more computers, such as multi-core computers, computers with graphics processing unit (GPU) clusters, large distributed computers, or clusters of computers with pooled hardware resources. The Session Boundary Controller (SBC) is used to manage or control the sessions of the terminal devices. The Signalling Business Unit (SBC) includes signaling plane and media plane functions. For example, it can receive media streams from managed terminal devices and send the received media streams to an AR media server. The AR media server processes the received uplink media streams to obtain downlink video streams. The uplink media streams include those from terminal devices; in multi-user video call scenarios, the uplink media streams may include those from multiple terminal devices. The downlink video streams can be sent by the AR media server to the corresponding terminal devices via the SBC. The terminal devices are devices with video call functionality, such as those equipped with cameras. Exemplarily, the terminal device can be a wearable device (e.g., a smartwatch), or a mobile phone, tablet, etc. This application does not impose special limitations on the specific form of the terminal device.

[0068] Figure 1 Taking three SBCs as an example, namely SBC1, SBC2, and SBC3, different SBCs manage different devices on different terminals. As an example, SBC1 manages the first terminal device, SBC2 manages the second terminal device, and SBC3 manages the third terminal device. Different terminal devices can also be managed by the same SBC; for example, SBC1 can also be used to manage a fifth terminal device. Figure 1(Not shown in the image). The SBC is deployed on edge nodes. Three SBCs are deployed on different edge nodes. The AR media server can be deployed on either edge nodes or the central node. Figure 1 Take the AR media server and SBC deployed on the edge node as an example. Figure 1 Taking three AMEs as an example, namely AME 1, AME 2, and AME 3. Edge nodes, compared to central nodes, are closer to end-user devices, providing these users with edge computing services, forwarding services, etc., reducing response latency and bandwidth costs, and alleviating the pressure on central nodes. Optionally, central nodes and edge nodes can be deployed in the cloud; in this case, central nodes can be called central cloud, and edge nodes can be called edge cloud. For distinction, in... Figure 1 In this configuration, SBC 1 and AME 1 are deployed in the first edge cloud, SBC 2 and AME 2 are deployed in the second edge cloud, and SBC 3 and AME 3 are deployed in the third edge cloud. The first edge cloud provides services to the first terminal device, the second edge cloud provides services to the second terminal device, and the third edge cloud provides services to the third terminal device. Exemplarily, the edge nodes can also be Mobile Edge Computing (MEC) nodes. It should be understood that an edge cloud can provide services to multiple terminal devices, meaning multiple terminal devices belong to a single edge cloud. Figure 1 Only three terminal devices belong to different edge clouds. As an example, the system also includes a fourth terminal device, which belongs to the first edge cloud (…). Figure 1 (Not shown in the image).

[0069] It should be noted that the functions of the AR media server and SBC can be deployed together or separately in the edge cloud, and this application embodiment does not limit this.

[0070] For example, the call system may also include an application server. The application server is used to establish a video call upon triggering by a terminal device. For instance, taking the video call between the first terminal device and the second terminal device as described above, the application server receives information from the first terminal device (such as call interface operation instructions), and sends the received information to the AR media server; thereby, the AR media server processes the uplink media stream based on the information from the first terminal device.

[0071] As an example, see Figure 1As shown, the application server may include a media plugin service function, also simply referred to as a plugin service or plugin server. The application server also includes an application service (AS). The media plugin service function is used to interact with terminal devices. In multi-user video call scenarios, it receives information or data triggered by multiple terminal devices, processes the received information or data accordingly to obtain different processed information or data for each terminal device, and then sends the processed information or data from each terminal device to the application service function. The application service function interacts with the AR media server, sending the processed data or information from the media plugin service function to the corresponding AR media server for each terminal device. The AR media server then processes the uplink media stream based on the processed data or information from the terminal devices sent by the media plugin service function.

[0072] It should be noted that the media plugin service function and AS can be deployed independently or jointly. For example, the media plugin service function and AS can be implemented through a single device or through one or more virtual machines.

[0073] In some embodiments, the central cloud can also deploy an IP multimedia subsystem (IMS) core network. The IMS core may include a call session control function (CSCF) and a home subscriber server (HSS). The IMS core may also include other network elements, which will not be elaborated upon in this embodiment. The CSCF is the call control center of the IMS core, implementing user access, authentication, session routing, and service triggering functions on the IP transmission platform. The CSCF may include one or more of the following: serving-call session control function (S-CSCF), proxy-CSCF (P-CSCF), and interrogating-CSCF (I-CSCF). The HSS is used to record user subscription data (such as user information and service data). See also... Figure 1 As shown, the SBC (including SBC1, SBC2 and SBC3) provides boundary control functions between the access network and the IMS core network, as well as between IMS core networks, and can provide access control, quality of service control and firewall traversal functions.

[0074] In some embodiments, an edge control (referred to as an edge plugin) can be deployed in the terminal device. The edge control is used to interact with the media plugin service function on the network side. The edge control can also establish an auxiliary transmission channel with the AR media enabler. This auxiliary transmission channel is used by the terminal device to send an auxiliary media stream to the AR media enabler or by the AR media enabler to send an auxiliary media stream to the terminal device. For example, the auxiliary media stream may include drawing point data of a user-drawn pattern. The auxiliary media stream may also include one or more of the following: point cloud data, spatial data (also referred to as spatial pose data), user-view video, or virtual models. Point cloud data refers to data recorded in the form of points; each point may include spatial location information, color information, or reflection intensity information, etc. Spatial data, also known as geometric data, is used to represent information such as the position, shape, size distribution, etc., of an object; it is a quantitative description of things and phenomena with locational significance existing in the real world. Virtual models may include one or more of the following: virtual human models, virtual object models, and source images (such as stickers, cartoon avatars, etc.), or virtual animation models, etc. User-perspective video, such as video captured by the user through the rear camera of the terminal device, or video captured by the front camera of the terminal device.

[0075] In this embodiment, the terminal device can establish different auxiliary transmission channels according to the type of auxiliary media stream being transmitted. For example, when point cloud data needs to be transmitted, auxiliary transmission channel 1 is established, and when drawing data needs to be transmitted, auxiliary transmission channel 2 is established. The auxiliary transmission channel 2 used for transmitting drawing data can also be called the Action channel, or other names may be used; this embodiment does not limit this. The terminal device can also transmit different types of auxiliary media streams through a single auxiliary transmission channel.

[0076] Optionally, the call system may also include a selective forwarding unit (SFU). The SFU is used to forward media streams from different edge clouds. The selective forwarding unit can be a forwarding server.

[0077] As an example, an AR media server can be deployed in a central cloud, see [link to relevant documentation]. Figure 2 As shown, this AR media server provides services to different terminal devices.

[0078] It should be noted that different terminal devices may have the same or different edge nodes. Additionally, different terminal devices may have the same or different center nodes. Different terminal devices may have the same or different AR media servers. Different terminals may have the same or different CSCFs.

[0079] The terminal devices involved in this application embodiment, such as the first terminal device to the third terminal device, all support video calls, such as VoLTE, Voice over New Radio (VoNR), or have social applications (APPs) installed. The terminal device can be a device that provides users with video recording and data connectivity. Examples include mobile phones (or "cellular" phones), smartphones, and portable, pocket-sized, handheld, wearable devices (such as smartwatches), tablets, personal computers (PCs), and in-vehicle computers.

[0080] Figure 3 A schematic diagram of an optional hardware structure for a terminal device is shown.

[0081] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, etc.

[0082] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0083] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0084] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0085] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0086] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0087] USB port 130 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal devices, and can also be used for data transfer between terminal devices and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0088] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the terminal device. In other embodiments of this application, the terminal device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0089] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal device. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0090] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0091] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0093] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on terminal devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0094] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0095] The wireless communication module 160 can provide solutions for wireless communication applications on terminal devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0096] In some embodiments, antenna 1 of the terminal device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the terminal device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0097] The terminal device implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0098] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device may include one or N1 displays 194, where N1 is a positive integer greater than 1.

[0099] Terminal devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0100] A touch sensor, also known as a "touch panel," can be located on the display screen 194. The touch sensor and display screen 194 together form a touchscreen, also called a "touch display." The touch sensor detects touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some embodiments, the touch sensor may also be located on the surface of the terminal device, in a different position than the display screen 194. The terminal device can implement pattern drawing functions using the touch sensor, GPU, display screen 194, and application processor.

[0101] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0102] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as a camera application), etc. The data storage area may store data created during the use of the terminal device (such as images captured by a camera), etc. Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the terminal device by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0103] The terminal device can implement audio functions such as music playback and recording through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor.

[0104] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The terminal device can receive button input and generate key signal inputs related to user settings and function control of the terminal device.

[0105] Motor 191 can generate vibration as a notification. Indicator 192 can be an indicator light, used to indicate charging status, power level changes, or to indicate messages, missed calls, notifications, etc.

[0106] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the terminal device. The terminal device can support one or N3 SIM card interfaces, where N3 is a positive integer greater than 1.

[0107] The solution provided in this application will be described in detail below with reference to specific embodiments.

[0108] Taking a video call between two terminal devices as an example, the two terminal devices are a first terminal device and a second terminal device. During the video call between the first terminal device and the second terminal device, the user of the first terminal device draws patterns to display the content to be explained. In this embodiment of the application, a layer for drawing patterns is overlaid on the video call interface, and this layer can be called a drawing layer. Of course, other names can also be used. The drawing layer interface can be transparent, white, or other colors; this embodiment of the application does not specifically limit this. See also Figure 4 The diagram shown is a schematic representation of a video call process according to an embodiment of this application. For ease of description, the user of the first terminal device will be referred to as User 1, and the user of the second terminal device will be referred to as User 2.

[0109] 401, The first terminal device collects the drawing points generated by the pattern drawn by user 1 to obtain drawing point data.

[0110] As an example, see Figure 5A The diagram illustrates a possible call interface of a first terminal device during a call between a first terminal device and a second terminal. The call interface includes a control bar 500. The control bar 500 includes a drawing control 501. The call interface may also include other controls, such as mute 502, speakerphone 503, and voice switch 504. The controls included in the control bar 500 of the call interface are merely an example. Furthermore, the position of the drawing control 501 on the call interface is not limited in this embodiment.

[0111] In response to the user 1's touch drawing control 501 operation on the first terminal device, the first terminal device overlays a drawing layer on the current display interface, such as... Figure 5B As shown. Figure 5B Taking the drawing layer as a whiteboard as an example, the image captured by the camera during a video call is covered after the drawing layer is overlaid. Figure 5B The example used here is a non-transparent whiteboard as the drawing layer and does not constitute a specific limitation. Furthermore, User 1 can draw patterns on the drawing layer. In response to User 1's drawing operation on the display screen of the first terminal device, the first terminal device collects the drawing points generated by User 1's drawing. The first terminal device can collect the coordinates of the touch points of User 1's hand on the display screen in real time at a preset collection rate to obtain drawing point data, which can also be called Action data. Specifically, the first terminal device continuously collects drawing points to obtain a collection sequence, and obtains drawing point data according to the order of the collection sequence. The collection rate is, for example, 30 drawing points per second. See also [example description missing]. Figure 5C The image shown is a schematic diagram of the display interface that includes drawing patterns. Figure 5C The circular ring in the image is only used as an example of a drawing point. In actual display, a black line is shown, and the collected drawing point is not displayed. The method for obtaining the drawing pattern will be described in detail later, and will not be described here.

[0112] It should be noted that user 1 of the first terminal device can determine the color of the drawn pattern, the style of the lines, the thickness of the lines, etc., according to their needs. After overlaying the drawing layer on the call interface, drawing tool controls can also be overlaid. As an example, in response to user 1's touch operation of drawing control 501, the drawing tool controls are displayed, for example, see [link to example]. Figure 5DAs shown, the drawing tool control includes selectable colors for the drawn pattern and the format of the lines. User 1 can select the line color and style of the drawn pattern according to their needs. For example, in addition to the coordinates of the drawn points, the drawing point data may also include the line color and color style selected by User 1. As an example, if User 1 does not select the line color or style of the drawn pattern, the drawing point data generated by the first terminal device may include the default line color and line style.

[0113] 402. The first terminal device encapsulates the collected drawing point data to obtain data packets belonging to the first data stream. In this embodiment, for ease of description, the data packets encapsulated from the drawing point data are referred to as drawing point data packets; however, other names can also be used, and this application does not specifically limit this. The following description will use the term "drawing point data packet" as an example. As an example, a drawing point data packet includes one drawing point data. The first terminal device continuously encapsulates drawing point data packets according to the order of the collected drawing point sequence to obtain the first data stream, which can also be called a drawing point data stream. The drawing point data stream consists of multiple consecutive drawing point data packets. This can be understood as real-time collection of drawing point data and real-time encapsulation into drawing point data packets; each collected drawing point data is encapsulated into a drawing point data packet and sent out.

[0114] For example, drawing point data packets can be done using the real-time transport protocol (RTP) or the real-time transport control protocol (RTCP).

[0115] 403, The first terminal device collects multimedia data generated during a video call between the first terminal device and the second terminal device. The multimedia data may be user audio data or user video data.

[0116] In this embodiment, the order of steps 401 and 403 is not limited. For example, drawing point data can be collected simultaneously with multimedia data collection.

[0117] 404. The first terminal device encapsulates the collected multimedia data to obtain data packets belonging to the second data stream. When the multimedia data includes audio data, the data packets of the second data stream can also be called audio data packets; when the multimedia data includes video data, the data packets of the second data stream can also be called video data packets. The data stream composed of multimedia data can be called a multimedia data stream. When the first terminal device sends multimedia data and drawing point data to the second terminal device, they are not sent together, but separately. Therefore, in order to ensure that the drawn pattern displayed at the receiving end is synchronized with the heard audio or seen video, the drawing point data and multimedia data need to be synchronized. Furthermore, it should be noted that the video data and audio data are also sent separately.

[0118] Figure 4 Taking multimedia data, specifically audio data, as an example, the first terminal device encapsulates audio data packets according to the order of the collected audio sequence to obtain an audio data stream.

[0119] It should be noted that during a call between the first terminal device and the second terminal device, video data may be generated simultaneously with audio data. Alternatively, video data may be generated without audio data, or audio data may be generated without video data. Synchronization of video and audio data can employ traditional synchronization methods, which will not be described in detail in this application. To ensure synchronization between the drawn pattern displayed on the receiving end, the played audio, and the displayed video, one approach is to synchronize the drawn point data and audio data; another approach is to synchronize the drawn point data and video data.

[0120] The following example demonstrates the synchronization of plotted data and audio data. Both audio and plotted data packets use RTP. During synchronization, timestamps or sequence numbers can be used to control the synchronization between plotted data and audio data.

[0121] For example, taking an audio sampling frequency of 8kHz as an example, the encapsulation period of the audio data packet is 0.02s, with one audio data packet encapsulated every 0.02s. For ease of distinction, the timestamp increment of the plotting point data packet is called the first timestamp increment, and the timestamp increment of the audio data packet is called the second timestamp increment. The second timestamp increment of two adjacent audio data packets is 160 = 0.02 * 8 * 1000. Plotting point data also belongs to video data, and the timestamp increment of the plotting point data is determined according to the method for determining the timestamp increment of video data. Taking a video sampling frequency of 90kHz as an example, the encapsulation period of the plotting point data packet is (1 / 30)s, that is, one plotting point data packet is encapsulated every (1 / 30)s, and 30 first data packets are encapsulated per second. The first timestamp increment of two adjacent plotting point data packets is 3000 = (1 / 30) * 90 * 1000. In the embodiments of this application, the period for acquiring plotting point data is called the acquisition period. When collecting drawing point data, the collection period is the same as the encapsulation period, which can be (1 / 30)s to collect one drawing point, that is, 30 drawing points are collected per second.

[0122] Audio data packets and drawing point data packets are sent based on the second timestamp increment of the audio data packets and the first timestamp increment of the drawing point data packets. For example, synchronization is determined by timestamp synchronization; drawing point data packets with the same timestamp or timestamps differing within a set range are synchronized with audio data packets. See [link to relevant documentation]. Figure 6 As shown, a plotting point data packet is sent for every 19 audio data packets. Figure 6 In the sequence number, n, n+1, n+2, ..., n+19, ..., n+38 represent the sequence number of the data packet. T, T+160, T+320, T+3040, T+3000, T+6000, T+6080 represent the timestamp of the data packet. Figure 6 The audio data packet with sequence number n corresponds synchronously to the drawing point data packet with sequence number n, and the audio data packet with sequence number n+19 corresponds synchronously to the drawing point data packet with sequence number n+1. That is, every 19 audio data packets correspond to one drawing point data packet, so the sequence number increment of the audio data packet when the drawing point data packet and the audio data packet are synchronized can be determined to be 19. From the perspective of sequence number, the sequence numbers of the synchronized audio data packets and the drawing point data packets can satisfy the relationship shown in Formula 1 below.

[0123] Nx = K*(N-N1) + N2 (Formula 1)

[0124] Where Nx represents the sequence number of the audio data packet, N represents the sequence number of the drawing point data packet, K represents the sequence number increment, and N1 and N2 represent the sequence numbers of the first synchronized drawing point data packet and audio data packet. Figure 6 In the process, the sequence number of the synchronous drawing point data packet and the sequence number of the audio data packet are both n, that is, N1 = N2 = n.

[0125] From a timestamp perspective, the data packets for drawing points with timestamp T+3040 and T+3000 are synchronized, and the audio data packets with timestamp T+6080 are synchronized with the data packets for drawing points with timestamp T+600. Therefore, the synchronization timestamp increment for the audio data packets is determined to be 3040, and the synchronization timestamp increment for the drawing point data packets is 3000. For ease of distinction, the synchronization timestamp increment for the audio data packets is referred to as the second synchronization timestamp increment, and the synchronization timestamp increment for the drawing point data packets is referred to as the first synchronization timestamp increment.

[0126] Therefore, in order for the receiving end to synchronously draw point data and audio data, the first terminal device can carry information for synchronizing the drawing point data and audio data in the drawing point data packet and audio data packet and send it to the device responsible for synchronizing the drawing point data and audio data. The information for synchronizing the drawing point data and audio data may include a first timestamp increment and a second timestamp increment, or it may include the video sampling frequency and the encapsulation period of the drawing point data packet used to determine the first timestamp increment, and the audio sampling frequency and the encapsulation period of the audio data packet used to determine the second timestamp increment, or it may include a sequence number increment, or it may include both the first synchronization timestamp increment and the second synchronization timestamp increment.

[0127] In this embodiment, the device responsible for synchronously drawing point data and audio data can be an AME, such as the AME to which the first terminal device belongs. The device responsible for synchronously drawing point data and audio data can also be the terminal device at the other end; in this embodiment, it is the second terminal device. Figure 4 In this example, the device responsible for synchronously drawing point data and audio data is taken as the terminal device at the other end. Further details will follow. Figure 9 The solution provided in this application embodiment will be described in detail using the device responsible for synchronously drawing point data and audio data as AME, without going into too much detail here.

[0128] In some embodiments, when the device responsible for synchronizing the drawing point data and audio data is a second terminal device, when the first terminal device sends the first timestamp increment and the second timestamp increment to the second terminal device, the first terminal device can carry the first timestamp increment in the drawing point data packet and the second timestamp increment in the audio data packet before sending them to the device responsible for synchronizing the drawing point data and audio data. For example, to reduce the transmission resources occupied by the first and second timestamps, the first terminal device can send the first timestamp increment once every first number of drawing point data packets and the second timestamp increment once every second number of audio data packets. Alternatively, the first terminal device can carry the first timestamp increment only in the first K1 drawing point data packets sent and the second timestamp increment only in the first K2 audio data packets sent.

[0129] In other embodiments, when the device responsible for synchronously drawing point data and audio data is a second terminal device, the first terminal device sends the video sampling frequency and the encapsulation period of the drawing point data packet used to determine the first timestamp increment, and the audio sampling frequency and the encapsulation period of the audio data packet used to determine the second timestamp increment to the second terminal device in the following manner:

[0130] The first terminal device can carry the video sampling frequency and the encapsulation period of the drawing point data packets in the drawing point data packets, and carry the audio sampling frequency and the encapsulation period of the audio data packets in the audio data packets, and send them to the device responsible for synchronizing the drawing point data and audio data. For example, to reduce the transmission resources occupied, the first terminal device can send the video sampling frequency and the encapsulation period of the drawing point data packets once every first number of drawing point data packets, and send the audio sampling frequency and the encapsulation period of the audio data packets once every second number of drawing point data packets. Alternatively, the first terminal device can carry the video sampling frequency and the encapsulation period of the drawing point data packets only in the first K1 drawing point data packets sent, and carry the audio sampling frequency and the encapsulation period of the audio data packets only in the first K2 audio data packets sent.

[0131] In some embodiments, when the device responsible for synchronously drawing point data and audio data is a second terminal device, when the first terminal device sends the sequence number increment to the second terminal device, the first terminal device can carry the sequence number increment in the audio data packet and send it to the device responsible for synchronously drawing point data and audio data. For example, to reduce the transmission resources occupied by the sequence number increment, the first terminal device can send the sequence number increment once every second number of audio data packets. Alternatively, the first terminal device can carry the sequence number increment only in the first K2 audio data packets sent.

[0132] In some embodiments, when the device responsible for synchronizing the drawing point data and audio data is a second terminal device, when the first terminal device sends the first synchronization timestamp increment and the second synchronization timestamp increment to the second terminal device, the first terminal device can carry the first synchronization timestamp increment in the drawing point data packet and the second synchronization timestamp increment in the audio data packet before sending them to the device responsible for synchronizing the drawing point data and audio data. For example, to reduce the transmission resources occupied by the first and second synchronization timestamp increments, the first terminal device can send the first synchronization timestamp increment once every first number of drawing point data packets and the second synchronization timestamp increment once every second number of audio data packets. Alternatively, the first terminal device can carry the first synchronization timestamp increment only in the first K1 drawing point data packets sent and the second synchronization timestamp increment only in the first K2 audio data packets sent.

[0133] 405, the first terminal device renders the user-drawn pattern based on the drawing point data and displays the user-drawn pattern, for example, see [link to example]. Figure 5C As shown; the first terminal device plays user audio data.

[0134] 406. The first terminal device sends the audio data stream and the drawing point data stream to AME1 via SBC1.

[0135] 407, AME1 sends the audio data stream and drawing point data stream to AME2 via SFU.

[0136] 408, AME2 sends the audio data stream and drawing point data stream to the second terminal device via SBC2.

[0137] 409. The second terminal device receives an audio data stream and a drawing point data stream from AME2. The second terminal device performs synchronization processing based on the time parameters of the audio data packets included in the audio data stream and the time parameters of the drawing point data packets included in the drawing point data stream. The time parameters may include timestamps and / or sequence numbers.

[0138] In one possible implementation, after the first terminal device and the second terminal device establish a video call, the time point of the first drawing point can be used as the starting point for the timestamp increment of the audio and drawing point data packets until the drawing is completed.

[0139] In some embodiments, the drawing point data packet carries a first timestamp increment, and the audio data packet carries a second timestamp increment.

[0140] The second terminal device performs synchronization processing based on the timestamps of the audio data packets in the audio data stream and the timestamps of the drawing point data packets in the drawing point data stream, a first timestamp increment, and a second timestamp increment. For example, the second terminal device can determine the second synchronization timestamp increment of the audio data packets and the first synchronization timestamp increment of the drawing point data packets based on the first and second timestamp increments, and then synchronize the audio data packets in the audio data stream and the drawing point data packets in the drawing point data stream based on these first and second synchronization timestamp increments. For instance, if the first timestamp increment is 160 and the second timestamp increment is 3000, the first synchronization timestamp increment can be 3000, and the second synchronization timestamp increment can be... Round refers to rounding to the nearest integer. That is, the timestamp interval between any two audio data packets synchronized with the drawing point data packet is 3040, and the timestamp interval between any two drawing point data packets synchronized with the audio data packet is 3000. For example, as shown in Figure 7, if the first audio data packet and the first drawing point data packet are synchronized, and both the timestamp of the first audio data packet and the first drawing point data packet are T, then the second synchronized audio data packet and drawing point data packet are the audio data packet with timestamp T+3040 and the drawing point data packet with timestamp T+3000, the third synchronized audio data packet and drawing point data packet are the audio data packet with timestamp T+6080 and the drawing point data packet with timestamp T+6000, and so on.

[0141] In other embodiments, the drawing point data packet carries a first synchronization timestamp increment, and the audio data packet carries a second synchronization timestamp increment. Further, the second terminal device synchronizes the audio data packet included in the audio data stream with the drawing point data packet included in the drawing point data stream according to the first synchronization timestamp increment and the second synchronization timestamp increment.

[0142] In some embodiments, the audio data packets carry a sequence number increment. The second terminal device performs synchronization processing based on the sequence numbers of the audio data packets included in the audio data stream, the sequence numbers of the drawing point data packets included in the drawing point data stream, and the sequence number increment. A number of audio data packets corresponding to each sequence number increment are synchronized with the drawing point data packets whose sequence numbers are incremented by 1. For example, the first audio data packet is synchronized with the first drawing point data packet, the second audio data packet is synchronized with the second drawing point data packet, the sequence number interval between the first and second audio data packets is the sequence number increment, and the sequence number interval between the first and second drawing point data packets is 1. For example, if the sequence number increment is K, then the correspondence between the sequence numbers of the synchronized audio data packets and the sequence numbers of the drawing point data packets is (K*(N-N1)+N2, N). For example, if the sequence number of the drawing point data packet is N, then the sequence number of the audio data packet synchronized with that drawing point data packet is K*(N-N1)+N2. Where N1 and N2 are the sequence numbers of the first synchronized drawing point data packet and audio data packet. As an example, the sequence number of the first plotting data packet and the synchronized audio data packet can be the same, i.e., N1 = N2. See also: Figure 7 The sequence number of the first drawing point data packet and the first audio data packet shown is n. The sequence number increment is 19. Then the audio data packet with sequence number n corresponds synchronously with the drawing point data packet with sequence number n, the audio data packet with sequence number n+19 corresponds synchronously with the drawing point data packet with sequence number n+1, and so on. That is, every 19 audio data packets correspond to 1 drawing point data packet.

[0143] In some embodiments, the audio data packets carry the audio sampling frequency and the encapsulation period of the audio data packets, and the drawing point data packets carry the video sampling frequency and the encapsulation period of the drawing point data packets. In one example, the second terminal device determines a second timestamp increment of the audio data packets based on the audio sampling frequency and the encapsulation period of the audio data packets, and determines a first timestamp increment of the drawing point data packets based on the video sampling frequency and the encapsulation period of the drawing point data packets, and performs synchronization processing on the audio data packets and the drawing point data packets based on the first timestamp increment and the second timestamp increment. In another example, the second terminal device determines a second timestamp increment of the audio data packets based on the audio sampling frequency and the encapsulation period of the audio data packets, and determines a first timestamp increment of the drawing point data packets based on the video sampling frequency and the encapsulation period of the drawing point data packets, further determining a sequence number increment of the audio data packets based on the first timestamp increment and the second timestamp increment, and then performs synchronization processing on the audio data packets and the drawing point data packets based on the sequence number increment.

[0144] 410. The second terminal device renders the drawing point data included in the drawing point data packet into a user-drawn pattern, and plays the audio data included in the audio data packet and displays the user-drawn pattern according to the synchronization processing result of the audio data packet and the drawing point data packet.

[0145] In one possible implementation, user 1 of the first terminal device draws a pattern on a whiteboard. In this scenario, the first terminal device may not send the corresponding video image of the whiteboard to other terminal devices. Therefore, the second terminal device only receives the drawing point data packet and the audio data packet, and thus the second terminal device executes step 401.

[0146] In another possible implementation, the first terminal device can annotate video images, such as in scenarios like remote guidance, advertising, and distance education. In this scenario, in addition to collecting drawing point data, the first terminal device also collects user video data and user audio data in step 403. The first terminal device sends video data streams, audio data streams, and drawing point data streams to other terminal devices. Further, the second terminal device receives the audio data stream, video data stream, and drawing point data stream. The second terminal device synchronizes the audio data stream, video data stream, and drawing point data stream to obtain a synchronization result. It should be noted that when performing synchronization, the audio data stream and drawing point data stream can be synchronized according to the synchronization method provided in the embodiments of this application, while the synchronization of the audio data stream and video data stream can be performed according to existing synchronization methods. Alternatively, the video data stream and drawing point data stream can be synchronized according to the synchronization method provided in the embodiments of this application, while the synchronization of the video data stream and audio data stream can be performed according to existing synchronization methods. After obtaining the synchronization processing result, the second terminal device renders the user video data from the video data stream to obtain the user video image and renders the drawing point data from the drawing point data stream to obtain the user-drawn pattern. The rendered user video image and the user-drawn pattern are then superimposed, or described as the user-drawn pattern superimposed on the user video image. For example, if video data packet 1 in the video data stream is synchronized with drawing point data packet 1 in the drawing point data stream, the user video image obtained by rendering the user video data packet 1 is superimposed with the user-drawn pattern obtained by rendering the drawing point data packet 1 to obtain the image to be displayed. Then, based on the synchronization processing result, the image to be displayed and the audio data included in the audio data stream are displayed.

[0147] In one possible implementation, during the process of the first terminal device sending the audio data stream and the drawing point data stream to the second terminal device, network transmission jitter may occur, leading to the loss of one or more data packets in the drawing point data stream. Furthermore, the loss of drawing point data packets can cause deviations in the user-drawn pattern rendered based on the received drawing data packets. See, for example... Figure 8 As shown, the user-drawn pattern displayed on the first terminal device is significantly different from the user-drawn pattern displayed on the second terminal device. Figure 8 In the diagram, black dots represent drawing points corresponding to lost drawing point data packets. Black rings represent drawing points corresponding to drawing point data actually collected by the first terminal device. To prevent significant deviations in user-drawn patterns due to lost drawing point data packets, this embodiment predicts lost drawing point data using data prediction, and then uses the predicted drawing point data to replace the lost drawing point data packets when drawing patterns.

[0148] In one example, when the second terminal device receives a drawing point data packet belonging to the drawing point data stream, it stores the drawing point data packet in a receive buffer. Further, when rendering the drawing point data packet into a user-drawn pattern, the second terminal device retrieves a first drawing point data packet from the receive buffer and performs a rendering operation on the drawing point data in the first drawing point data packet. If a second drawing point data packet belonging to the drawing point data stream is not retrieved from the receive buffer at the rate at which the second terminal device renders the drawing points of the user-drawn pattern, the second drawing point data packet is predicted based on the first N drawing point data packets that have already been rendered. This second drawing point data packet is the next drawing point data packet adjacent to the first drawing point data packet in the drawing point data stream, or it can be described as having a sequence number adjacent to the sequence number of the first drawing point data packet, and the sequence number of the second drawing point data packet is greater than the sequence number of the first drawing point data packet; or it can be described as having a timestamp greater than the timestamp of the first drawing point data packet, and the timestamp of the second drawing point data packet is spaced apart from the timestamp of the first drawing point data packet by a first timestamp increment. Further still, the second terminal device renders the second drawing point data based on the predicted second drawing point data packet.

[0149] It should be understood that when rendering the drawing point data in this embodiment, pattern data to be displayed is generated. The pattern data to be displayed is used to describe the connection between two drawing points. For example, taking the rendering of the second drawing point data as an example, the second terminal device renders the second drawing point data to generate pattern data that represents the connection between the first drawing point described by the first drawing point data and the second drawing point described by the second drawing point data. Then, the generated pattern data is displayed.

[0150] Specifically, the user-drawn pattern can be rendered based on the line color, line style, line thickness, and other information included in the drawing point data.

[0151] In another example, after the second terminal device renders the second drawing point data based on the predicted second drawing point data packet, when it receives a second drawing point data packet from the first terminal device, it can replace the first pattern data generated by rendering the predicted second drawing point data with second pattern data generated from the second drawing point data included in the received second drawing point data packet. Furthermore, the second pattern data is displayed.

[0152] The above Figure 4 The corresponding embodiment is described from the perspective of the receiving terminal device being responsible for the synchronous processing and rendering operations of the point data stream and audio data stream. In this embodiment, the AME deployed on the cloud side can also be responsible for the synchronous processing of the point data stream and audio data stream, as well as the rendering operations of the point data. The following is in conjunction with... Figure 9This document details the process of synchronizing the drawing point data stream and audio data stream, as well as rendering the drawing point data, performed by AME. This embodiment uses an IMS architecture as an example, combined with... Figure 1 The example shown is a communication system based on the IMS architecture.

[0153] See Figure 9 The diagram shown is a flowchart of a video call method provided in an embodiment of this application. Figure 9 The example also uses a video call between two terminal devices, designated as Terminal 1 and Terminal 2. During the video call, the user on Terminal 1 draws patterns to illustrate the content being explained.

[0154] 901-904, see 401-404, will not be repeated here.

[0155] 905, the first terminal device sends the audio data stream and the drawing point data stream to AME1 via SBC1.

[0156] Figure 9 In this example, we take the device responsible for synchronously drawing point data and audio data as AME. The AME responsible for synchronously drawing point data and audio data can be the AME of the sending terminal device, i.e., AME1. AME1 renders the point data to obtain the pattern data to be displayed, and then merges the pattern data to be displayed with the data to be played obtained from rendering the audio data before sending it to AME2.

[0157] To enable AME1 to synchronously draw point data and audio data, the first possible approach is for the first terminal device to include information for synchronizing point data and audio data in the point data packet and audio data packet and send it to AME1. The second possible approach is for the first terminal device to send this information to the application server, which then forwards it to AME1.

[0158] Information used to synchronize plotting point data and audio data may include a first timestamp increment and a second timestamp increment, or may include a video sampling frequency and plotting point data packet encapsulation period used to determine the first timestamp increment, and an audio sampling frequency and audio data packet encapsulation period used to determine the second timestamp increment, or may include a sequence number increment, or may include a first synchronization timestamp increment and a second synchronization timestamp increment.

[0159] In some embodiments of the first possible approach, the first terminal device may carry the first timestamp increment in the drawing point data packet and the second timestamp increment in the audio data packet before sending them to AME1. For example, to reduce the transmission resources occupied by the first and second timestamps, the first terminal device may send the first timestamp increment once every first number of drawing point data packets and the second timestamp increment once every second number of drawing point data packets. Alternatively, the first terminal device may carry the first timestamp increment only in the first K1 drawing point data packets sent and the second timestamp increment only in the first K2 audio data packets sent.

[0160] In other embodiments, the first terminal device may carry the video sampling frequency and the encapsulation period of the drawing point data packet in the drawing point data packet, and carry the audio sampling frequency and the encapsulation period of the audio data packet in the audio data packet and send them to AME1.

[0161] In other embodiments, the first terminal device may send the serial number increment in the audio data packet to AME1.

[0162] In some other embodiments, the first terminal device may carry the first synchronization timestamp increment in the drawing point data packet and carry the second synchronization timestamp increment in the audio data packet and send it to AME1.

[0163] When using the first possible approach, details regarding how the information used to synchronize plot point data and audio data is carried in the plot point data packet and audio data packet can be found in [link to relevant documentation]. Figure 4 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0164] In some embodiments of the second possible approach, the first terminal device may send the first timestamp increment and the second timestamp increment to the application server, which in turn sends the first timestamp increment and the second timestamp increment to AME1. For example, the first terminal device may send the first timestamp increment and the second timestamp increment to the application server during the process of establishing a call connection with the second terminal device, which in turn sends the first timestamp increment and the second timestamp increment to AME1. Alternatively, the first terminal device may send the first timestamp increment and the second timestamp increment to the application server after completing the call connection with the second terminal device, which in turn sends the first timestamp increment and the second timestamp increment to AME1.

[0165] In other embodiments, the first terminal device can send the video sampling frequency and the encapsulation period of the drawing point data packet, the audio sampling frequency and the encapsulation period of the audio data packet to the application server, and then the application server sends the video sampling frequency and the encapsulation period of the drawing point data packet, the audio sampling frequency and the encapsulation period of the audio data packet to AME1.

[0166] In some other embodiments, the first terminal device may send the serial number increment to the application server, which in turn sends it to AME1.

[0167] In some other embodiments, the first terminal device may send the first synchronization timestamp increment and the second synchronization timestamp increment to the application server, and then the application server may send the first synchronization timestamp increment and the second synchronization timestamp increment to AME1.

[0168] 906, AME1 performs synchronization processing based on the time parameters of the audio data packets included in the audio data stream and the time parameters of the drawing point data packets included in the drawing point data stream. The time parameters may include timestamps and / or sequence numbers.

[0169] In some embodiments, AME1 performs synchronization processing based on the timestamps of the audio data packets included in the audio data stream and the timestamps, first timestamp increments, and second timestamp increments of the drawing point data packets included in the drawing point data stream. The methods for obtaining the first and second timestamp increments can be found in the relevant description of 905, and will not be repeated here. For example, the first timestamp increment is carried in the drawing point data packet, and the second timestamp increment is carried in the audio data packet. Another example is that AME1 receives the first and second timestamp increments sent by the first terminal device through the application server.

[0170] For example, AME1 can determine the second synchronization timestamp increment of the audio data packet and the first synchronization timestamp increment of the drawing point data packet based on the first and second timestamp increments. Then, it synchronizes the audio data packets in the audio data stream with the drawing point data packets in the drawing point data stream based on these first and second synchronization timestamp increments. This method of AME1 synchronizing the audio data packets in the audio data stream with the drawing point data packets in the drawing point data stream based on these first and second synchronization timestamp increments is similar to... Figure 4 In the corresponding embodiment, the second terminal device performs synchronization in a similar manner based on the first synchronization timestamp increment and the second synchronization timestamp increment, see [link to relevant documentation]. Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0171] In other embodiments, AME1 obtains a first synchronization timestamp increment and a second synchronization timestamp increment. AME1 synchronizes the audio data packets included in the audio data stream with the drawing point data packets included in the drawing point data stream based on the first synchronization timestamp increment and the second synchronization timestamp increment. The method for obtaining the first synchronization timestamp increment and the second synchronization timestamp increment can be found in the relevant description of 905, and will not be repeated here.

[0172] In some embodiments, AME1 obtains the sequence number increment and then performs synchronization processing based on the sequence numbers of the audio data packets included in the audio data stream, the sequence numbers of the drawing point data packets included in the drawing point data stream, and the sequence number increment. The method for obtaining the sequence number increment can be found in the relevant description of 905, and will not be repeated here. The method by which AME1 performs synchronization processing based on the sequence numbers of the audio data packets included in the audio data stream, the sequence numbers of the drawing point data packets included in the drawing point data stream, and the sequence number increment is similar to... Figure 4 In the corresponding embodiment, the second terminal device performs synchronization based on the serial number in a similar manner, see [link to relevant documentation]. Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0173] In some embodiments, AME1 obtains the audio sampling frequency and the encapsulation period of the audio data packet, and the drawing point data packet carries the video sampling frequency and the encapsulation period of the drawing point data packet. In one example, AME1 determines a second timestamp increment of the audio data packet based on the audio sampling frequency and the encapsulation period of the audio data packet, and determines a first timestamp increment of the drawing point data packet based on the video sampling frequency and the encapsulation period of the drawing point data packet, and performs synchronization processing on the audio data packet and the drawing point data packet based on the first timestamp increment and the second timestamp increment. In another example, AME1 determines a second timestamp increment of the audio data packet based on the audio sampling frequency and the encapsulation period of the audio data packet, and determines a first timestamp increment of the drawing point data packet based on the video sampling frequency and the encapsulation period of the drawing point data packet, further determining a sequence number increment of the audio data packet based on the first timestamp increment and the second timestamp increment, and then performs synchronization processing on the audio data packet and the drawing point data packet based on the sequence number increment.

[0174] 907, AME1 renders the drawing point data included in the drawing point data packet based on the synchronization processing results to obtain the user-drawn pattern to be displayed.

[0175] Optionally, the user-drawn pattern to be displayed can be encapsulated in an RTP packet or an RTCP packet and sent to the second terminal device.

[0176] In one possible implementation, when user 1 of the first terminal device draws a user pattern on a whiteboard, in this scenario, the first terminal device may not send the corresponding video image of the whiteboard to other terminal devices. Therefore, AME1 only receives the drawing point data packet and the audio data packet, and AME1 executes step 906. During rendering, AME1 replaces the pixel values ​​of the corresponding positions on the whiteboard with the pixel values ​​of the drawing points included in the drawing point data packet. Alternatively, it can be described as replacing the pixel values ​​of the corresponding positions in the previously rendered image with the pixel values ​​of the drawing points included in the currently received drawing point data packet. The timestamp or sequence number of the data packet encapsulating the user-drawn pattern to be displayed can be the timestamp or sequence number of the drawing point data packet. AME1 may also re-determine the timestamp or sequence number for the data packet and audio data packet encapsulating the user-drawn pattern to be displayed based on the synchronized audio data packet and drawing point data packet; this application does not specifically limit this.

[0177] In another possible implementation, the first terminal device can annotate video images, such as in scenarios like remote guidance, advertising, and distance education. In this scenario, in addition to collecting drawing point data, the first terminal device also collects user video data and user audio data in step 403. The first terminal device sends video data streams, audio data streams, and drawing point data streams to other terminal devices. Further, AME1 receives the audio data stream, video data stream, and drawing point data stream. AME1 synchronizes the audio data stream, video data stream, and drawing point data stream to obtain a synchronization result. It should be noted that when performing synchronization, the audio data stream and drawing point data stream can be synchronized according to the synchronization method provided in the embodiments of this application, while the synchronization of the audio data stream and video data stream can be performed according to existing synchronization methods. Alternatively, the video data stream and drawing point data stream can be synchronized according to the synchronization method provided in the embodiments of this application, while the synchronization of the video data stream and audio data stream can be performed according to existing synchronization methods. After obtaining the synchronization processing result, AME1 renders the user video data from the video data stream to obtain the user video image and renders the drawing point data from the drawing point data stream to obtain the user drawing pattern. The rendered user video image and the user drawing pattern are then superimposed, or in other words, the user drawing pattern is superimposed on the user video image. Specifically, the pixel values ​​of the drawing points included in the drawing point data packet replace the pixel values ​​of the corresponding positions in the user video image. For example, if video data packet 1 in the video data stream is synchronized with drawing point data packet 1 in the drawing point data stream, the user video image obtained by rendering the user video data packet 1 and the user drawing pattern obtained by rendering the drawing point data packet 1 are superimposed to obtain superimposed video data. As an example, AME1 can encapsulate the superimposed video data into data packets to obtain a superimposed video stream. The data packets obtained by encapsulating the superimposed video data can use RTP or RTCP format. The superimposed data packets can use the timestamp or sequence number of video data packet 1, or the timestamp or sequence number of drawing point data packet 1. AME1 can also re-determine the timestamp or sequence number for the superimposed data packets and audio data packets based on the synchronized audio data packets and the superimposed data packets; this application does not specifically limit this.

[0178] 908, AME1 sends the user-drawn pattern to be displayed to the first terminal device, and sends the user-drawn pattern and audio data packet to AME2.

[0179] 909, AME2 sends user-drawn patterns and audio data packets to the second terminal device via SFU and AME2.

[0180] 910, the first terminal device displays the user-drawn pattern to be displayed.

[0181] 911, the second terminal device displays the user-drawn pattern to be displayed and plays the user audio data included in the audio data packet. As an example, before displaying the user-drawn pattern to be displayed and playing the audio data included in the audio data packet, the second terminal device may perform synchronization processing according to the time parameters of the data packet encapsulating the user-drawn pattern and the audio data packet. The synchronization processing method is similar to the synchronization processing method used in AME1, and this application does not make specific limitations.

[0182] In one possible implementation, during the process of the first terminal device sending the audio data stream and the drawing point data stream to AME1, network transmission jitter may occur, leading to the loss of one or more data packets in the drawing point data stream. Furthermore, the loss of drawing point data packets can cause deviations in the user-drawn pattern rendered based on the received drawing data packets. See, for example... Figure 8 As shown. In order to prevent significant deviations in user-drawn patterns due to the loss of drawing point data packets, this embodiment of the application predicts the lost drawing point data through data prediction, and then uses the predicted drawing point data to replace the lost drawing point data packets when drawing the pattern.

[0183] Furthermore, during the process of the first terminal device sending the audio data stream and the drawing point data stream to AME1, network transmission fluctuations may occur, leading to pulsed arrival of drawing point data packets in the drawing point data stream. Further, the delayed arrival of drawing point data packets can cause the drawing point data packets in the receiving buffer to be insufficient to support the rendering rate. The rendering rate of AME1 is the same as the rate at which the first terminal device acquires drawing points. In this embodiment, delayed drawing point data is predicted using a data prediction method to ensure that the rendering rate of drawing point data is the same as the acquisition rate of drawing points. Therefore, when drawing a pattern, the predicted drawing point data is rendered first, and when delayed drawing point data is received subsequently, the predicted drawing point data with deviations can be corrected.

[0184] In one example, when AME1 receives a drawing point data packet belonging to the drawing point data stream, it stores the drawing point data packet in the receive buffer. Further, when AME1 renders the drawing point data packet into a user-drawn pattern, it retrieves the first drawing point data packet from the receive buffer and performs a rendering operation on the drawing point data in the first drawing point data packet. If, at the rate at which AME1 renders the drawing points of the user-drawn pattern, a second drawing point data packet belonging to the drawing point data stream is not retrieved from the receive buffer, the second drawing point data packet is predicted based on the first N drawing point data packets that have already been rendered. This second drawing point data packet is the next drawing point data packet adjacent to the first drawing point data packet in the drawing point data stream, or it can be described as having a sequence number adjacent to the sequence number of the first drawing point data packet, and the sequence number of the second drawing point data packet is greater than the sequence number of the first drawing point data packet; or it can be described as having a timestamp greater than the timestamp of the first drawing point data packet, and the timestamp of the second drawing point data packet is spaced apart from the timestamp of the first drawing point data packet by a first timestamp increment. Further still, AME1 renders the second drawing point data based on the predicted second drawing point data packet.

[0185] In another example, after AME1 renders the predicted second drawing point data, when it receives a second drawing point data packet from the first terminal device, it can replace the first pattern data generated by the predicted second drawing point data with the second pattern data generated by the second drawing point data included in the received second drawing point data packet. Optionally, when the time elapsed between receiving the second drawing point data packet from the second terminal device and the time elapsed between receiving the predicted second drawing point data packet reaches a duration threshold, the second pattern data generated by the second drawing point data included in the received second drawing point data packet can be discarded instead of replacing the first pattern data generated by the predicted second drawing point data. When the second drawing point data included in the received second drawing point data packet is the same as the predicted second drawing point data, or when the distance between the drawing point indicated by the second drawing point data included in the received second drawing point data packet and the drawing point indicated by the predicted second drawing point data is less than a distance threshold, the received second drawing point data packet is discarded.

[0186] In another possible implementation, if network transmission fluctuations cause multiple drawing point data packets belonging to the drawing point data stream to arrive instantaneously, AME1, when rendering the drawing point data in the drawing point data packets, still renders the received drawing point data at the rate of rendering the drawing points of the user-drawn pattern. For example, the multiple drawing point data packets received instantaneously include both drawing point data packets that have already undergone prediction and drawing point data packets that arrive newly in the order of reception. AME1 can discard the drawing point data packets that have already undergone prediction and then render the newly received drawing point data packets at the rate of rendering the drawing points of the user-drawn pattern.

[0187] In another possible implementation, the first terminal device sends the drawing point data to AME1, which then performs a rendering operation and sends the rendered drawing point data to obtain the user-drawn pattern back to the first terminal device, which then displays the user-drawn pattern. Due to the uplink and downlink time delay, a deviation occurs between the displayed user pattern and the actual user-drawn pattern. For example, there may be a break between the end of the displayed user-drawn line and the actual touch point of the user, see, for example... Figure 10 As shown, this affects the rendering effect. To improve the user experience, this embodiment of the application adopts an edge-cloud linkage rendering method, which can use the rendering point data collected by the rendering party's terminal device to render and fill the delayed content.

[0188] In one example, after the first terminal device collects audio data and drawing point data, it performs rendering based on the drawing point data to obtain a user-drawn pattern and displays the rendered user-drawn pattern. Then, upon receiving a user-drawn pattern from AME1, it displays the user-drawn pattern from AME1. When the first terminal device performs the rendering operation based on the drawing point data, the line color and / or style used to render the user pattern can use a default line color and / or style. Based on this, user 1 can determine the user patterns displayed by each video call party based on the displayed user patterns. Furthermore, user 1 can adjust the speed at which the content of the user pattern is described, enabling each video call party to accurately understand the content described by user 1, thus improving the user experience.

[0189] In another example, after the first terminal device collects audio data and drawing point data, it performs rendering based on the drawing point data to obtain a user-drawn pattern and displays the rendered user-drawn pattern. Then, upon receiving a user-drawn pattern from AME1, it can discard the user-drawn pattern from AME1.

[0190] In another example, after the first terminal device collects audio data and drawing point data, it performs rendering based on the drawing point data to obtain a user-drawn pattern and displays the rendered user-drawn pattern. Correspondingly, when AME1 synchronously sends the user-drawn pattern to be displayed and the data to be played to AME2, it no longer needs to send the user-drawn pattern to be displayed to the first terminal device. This prevents a break between the end of the displayed user-drawn line and the user's actual touch point due to uplink and downlink delays.

[0191] Based on the same inventive concept as the method embodiments, this application provides a communication device 1100, the structure of which is as follows: Figure 11 As shown, the system includes a communication unit 1101 and a processing unit 1102. Exemplarily, the communication unit 1101 can be used to both transmit and receive signals. Optionally, the communication unit 1101 may include a transmitting unit and a receiving unit, whereby the transmitting unit transmits signals and the receiving unit receives signals.

[0192] In one possible application scenario, the communication device 1100 is applied to a media server, specifically to implement the method executed by the media server (e.g., any one of AME1-AME2) in the above method embodiments.

[0193] The communication unit 1101 is used to receive a first data stream from a first terminal device, wherein the data packets in the first data stream include drawing point data of a user-drawn pattern collected by the first terminal device.

[0194] The communication unit 1101 is also configured to receive a second data stream from the first terminal device, wherein the data packets in the second data stream include user audio data and / or user video data of the first terminal device;

[0195] Processing unit 1102 is used to synchronize the first data stream and the second data stream according to the time parameters of the data packets included in the first data stream and the time parameters of the data packets included in the second data stream to obtain a synchronization processing result;

[0196] When the data packets in the second data stream include user audio data, the processing unit 1102 is further configured to render the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user drawing pattern, and send the user drawing pattern and the second data stream to the second terminal device; or,

[0197] When the data packets in the second data stream include user video data, the processing unit 1102 is further configured to render the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user drawing pattern, and render the user video data in the data packets of the second data stream to obtain a user video image, superimpose the user drawing pattern on the user video image to obtain a third video stream, and send the third video stream to the second terminal device.

[0198] In one possible implementation, the media server renders the drawing points of the user-drawn pattern at the same rate as the first terminal device collects the drawing points of the user-drawn pattern.

[0199] In one possible implementation, when the sampling frequency of the first data stream is different from the sampling frequency of the second data stream, the processing unit 1102 is specifically used for:

[0200] When the time parameter includes a timestamp, the data packets of the first data stream and the data packets of the second data stream are synchronized based on the timestamps and first timestamp increments of the data packets of the first data stream, the timestamps and second timestamp increments of the data packets of the second data stream, wherein the first timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the first data stream, and the second timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the second data stream; or,

[0201] When the time parameter includes a sequence number, the data packets of the first data stream and the data packets of the second data stream are synchronized according to the sequence number of the data packets of the first data stream, the sequence number of the data packets of the second data stream, and the sequence number increment, wherein the sequence number increment is determined according to the sampling frequency of the first data stream and the sampling frequency of the second data stream.

[0202] In one possible implementation, the processing unit 1102 is further configured to store the received data packets belonging to the first data stream in the receive buffer of the media server. When the processing unit 1102 executes the rendering of the drawing point data in the data packets of the first data stream to obtain a user-drawn pattern, specifically, it is configured to retrieve the first data packet belonging to the first data stream from the receive buffer and render the first data packet as the first drawing point of the user-drawn pattern; if a second data packet belonging to the first data stream is not retrieved from the receive buffer at the rate at which the media server renders the drawing points of the user-drawn pattern, it predicts the drawing point data of the second data packet based on the first N data packets that have already been rendered.

[0203] Wherein, the sequence number of the second data packet is adjacent to that of the first data packet and the sequence number of the second data packet is greater than that of the first data packet or the time interval between the timestamp of the second data packet and the timestamp of the first data packet is a time threshold, wherein the time threshold is determined according to the sampling frequency of the first data stream and the sampling frequency of the second data stream;

[0204] The predicted drawing point data of the second data packet is rendered as the second drawing point of the user-drawn pattern.

[0205] In one possible implementation, the communication unit 1101 is further configured to receive the second data packet after the processing unit 1102 renders the predicted drawing point data of the second data packet into the second drawing point of the user drawing pattern; the processing unit 1102 is further configured to replace the second drawing point with the drawing point rendered according to the drawing point data of the received second data packet.

[0206] In one possible implementation, the first data stream is a Real-time Streaming Transport Protocol (RTP) stream, and the second data stream is an RTP stream; or, the first data stream is a Real-time Streaming Transport Control Protocol (RTCP) stream, and the second data stream is an RTCP stream.

[0207] Based on the same inventive concept as the method embodiments, this application provides a communication device 1200, the structure of which is as follows: Figure 12 As shown, the device includes a processing unit 1201, a communication unit 1202, and a display unit 1203. Exemplarily, the communication unit 1202 can be used to both transmit and receive signals. Optionally, the communication unit 1202 may include a transmitting unit and a receiving unit, whereby the transmitting unit transmits signals and the receiving unit receives signals. This communication device 1200 is used in a terminal device, specifically to implement the method executed by the terminal device (e.g., the first terminal device) in the above method embodiments.

[0208] In response to the drawing operation of the user drawing pattern on the first terminal device, the processing unit 1201 collects the drawing points of the user drawing pattern in real time to obtain drawing point data.

[0209] As an example, the device 1200 may also include a first acquisition unit 1204. The processing unit 1201 acquires drawing point data through the first acquisition unit 1204.

[0210] The processing unit 1201 collects multimedia data in real time, including user audio data or user video data of the first terminal device.

[0211] The device 1200 may further include a second acquisition unit 1205 or a third acquisition unit 1206. The processing unit 1201 acquires user audio data through the second acquisition unit 1205. The processing unit 1201 acquires user video data through the third acquisition unit 1206.

[0212] The processing unit 1201 is further configured to encapsulate the drawing point data of the user-drawn pattern acquired in real time into a data packet belonging to a first data stream, and to encapsulate the multimedia data acquired in real time into a data packet belonging to a second data stream. The data packet of the first data stream includes the drawing point data of the user-drawn pattern, and the data packet of the second data stream includes user audio data or user video data of the first terminal device. The data packet in the first data stream carries a first timestamp, and the data packet in the second data stream carries a second timestamp. The first timestamp and the second timestamp are determined based on the sampling frequency of the drawing points of the user-drawn pattern and the sampling frequency of the user video data or user audio data encapsulated in the second data packet.

[0213] The communication unit 1203 sends the first data stream and the second data stream to the media server through the session boundary controller (SBC).

[0214] In one possible implementation, the processing unit 1201 is further configured to render the drawing point data to obtain a user-drawn pattern and display the user-drawn pattern.

[0215] In one possible implementation, the communication unit 1202 is further configured to receive a first user-drawn pattern from the media server via the SBC, the first user-drawn pattern being obtained by the media server rendering drawing point data in the data packets of the first data stream;

[0216] The display unit 1203 is used to display the pattern drawn by the first user.

[0217] In one possible implementation, the processing unit 1201 is further configured to render the drawing point data to obtain a second user drawing pattern and display the second user drawing pattern before the communication unit 1202 receives the first user drawing pattern sent by the media server through the first SBC.

[0218] The display unit 1203 is further configured to display the first user-drawn pattern after the communication unit 1202 receives the first user-drawn pattern from the media server through the SBC, wherein the displayed first user-drawn pattern covers the second user-drawn pattern.

[0219] In one possible implementation, the first data stream is a Real-time Streaming Transport Protocol (RTP) stream, and the second data stream is an RTP stream; or, the first data stream is a Real-time Streaming Transport Control Protocol (RTCP) stream, and the second data stream is an RTCP stream.

[0220] Based on the same inventive concept as the method embodiments, this application provides a communication device 1300, the structure of which is as follows: Figure 13 As shown, the device includes a processing unit 1301, a communication unit 1302, a display unit 1303, and a playback unit 1304. Exemplarily, the communication unit 1302 can be used to both send and receive signals. Optionally, the communication unit 1302 may include a sending unit and a receiving unit, whereby the sending unit sends signals and the receiving unit receives signals. This communication device 1300 is used in a terminal device, specifically to implement the methods executed by the terminal device (e.g., a second terminal device) in the above method embodiments.

[0221] Communication unit 1302 is used to receive a first data stream from a media server, wherein the data packets in the first data stream include drawing point data of a user-drawn pattern collected by a first terminal device;

[0222] The communication unit 1302 is also configured to receive a second data stream from a media server, wherein the data packets of the second data stream include user audio data and / or user video data of the first terminal device;

[0223] Processing unit 1301 synchronizes the first data stream and the second data stream according to the time parameters of the data packets in the first data stream and the time parameters of the data packets in the second data stream to obtain a synchronization processing result;

[0224] When the data packets in the second data stream include user audio data, the processing unit 1301 renders the data packets in the first data stream according to the synchronization processing result to obtain a user-drawn pattern, the display unit 1303 displays the user-drawn pattern, and the playback unit 1304 plays the user audio data; or...

[0225] When the data packets in the second data stream include user video data, the processing unit 1301 renders the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user drawing pattern, and renders the user video data in the data packets of the second data stream to obtain a user video image, and superimposes the user drawing pattern on the user video image, and the display unit 1303 displays the user drawing pattern superimposed on the user video image.

[0226] In one possible implementation, the rate at which the second terminal device renders the drawing points of the user-drawn pattern is the same as the rate at which the first terminal device acquires the drawing points of the user-drawn pattern.

[0227] In one possible implementation, when the sampling frequency of the first data stream is different from that of the second data stream, the processing unit 1301, when synchronizing the first data stream and the second data stream according to the time parameters of the data packets in the first data stream and the time parameters of the data packets in the second data stream, specifically performs the following:

[0228] When the time parameter includes a timestamp, the data packets in the first data stream and the data packets in the second data stream are synchronized based on the timestamps and first timestamp increments of the data packets in the first data stream, the timestamps and second timestamp increments of the data packets included in the second data stream, wherein the first timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the first data stream, and the second timestamp increment is the difference between the timestamps of two adjacent data packets belonging to the second data stream; or,

[0229] When the time parameter includes a sequence number, the data packets in the first data stream and the data packets in the second data stream are synchronized based on the sequence number of the data packets in the first data stream, the sequence number of the data packets in the second data stream, and the sequence number increment. The sequence number increment is determined based on the sampling frequency of the first data stream and the sampling frequency of the second data stream.

[0230] In one possible implementation, the processing unit 1301 stores the received data packets belonging to the first data stream in a receive buffer;

[0231] When the processing unit 1301 renders the drawing point data of the data packet of the first data stream to obtain the user-drawn pattern, it is specifically used for:

[0232] The first data packet belonging to the first data stream is obtained from the receiving buffer, and the drawing point data of the first data packet is rendered as the first drawing point of the user drawing pattern; if the second data packet belonging to the first data stream is not obtained from the receiving buffer according to the rate at which the second terminal device renders the drawing points of the user drawing pattern, the drawing point data of the second data packet is predicted according to the first N data packets belonging to the first data stream that have been rendered.

[0233] Wherein, the sequence number of the second data packet is adjacent to that of the first data packet and the sequence number of the second data packet is greater than that of the first data packet or the time interval between the timestamp of the second data packet and the timestamp of the first data packet is a time threshold, wherein the time threshold is determined according to the sampling frequency of the first data stream and the sampling frequency of the second data stream;

[0234] The predicted drawing point data of the second data packet is rendered as the second drawing point of the user-drawn pattern.

[0235] In one possible implementation, the communication unit 1302 receives the second data packet after the processing unit 1301 renders the predicted drawing point data of the second data packet into the second drawing point of the user-drawn pattern. The processing unit 1301 then replaces the second drawing point with the drawing point rendered according to the received second data packet.

[0236] In one possible implementation, the first data stream is a Real-time Streaming Transport Protocol (RTP) stream, and the second data stream is an RTP stream; or, the first data stream is a Real-time Streaming Transport Control Protocol (RTCP) stream, and the second data stream is an RTCP stream.

[0237] The unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into a single processor, exist as separate physical units, or two or more units can be integrated into a single module. The integrated units described above can be implemented in hardware or as software functional modules.

[0238] In the embodiments of this application, both the media server and the terminal device can be presented in an integrated manner, divided into various functional modules. Here, "module" can refer to a specific ASIC, circuit, processor and memory executing one or more software or firmware programs, integrated logic circuit, and / or other devices that can provide the above functions.

[0239] In a simplified embodiment, the SBC, media server, or application server may employ... Figure 14 As shown in the figure.

[0240] like Figure 14 The illustrated device 1400 includes at least one processor 1401 and a communication interface 1402. Optionally, it may also include a memory 1403.

[0241] In one possible implementation, when the media server adopts Figure 14 When in the form shown, Figure 14 The processor 1401 can invoke computer execution instructions stored in the memory 1403, enabling the media server to execute the method described in any of the above method embodiments.

[0242] The processor 1401 can communicate with other devices through the communication interface 1402. For example, the processor 1401 can receive a first data stream and a second data stream from a first terminal device through the communication interface 1402, and send the first data stream and the second data stream to the second terminal device, or send user-drawn patterns and audio data streams.

[0243] In one example, memory 1403 stores information for implementing... Figure 11 The functions of the communication unit 1101 and the processing unit 1102 in the computer are to execute instructions. Figure 14 The functions / implementation processes of the communication unit 1101 and the processing unit 1102 can be achieved through... Figure 14 The processor 1401 in the memory calls computer execution instructions stored in memory 1403 and uses communication interface 1402 to achieve this. In another example, Figure 11 The function of the communication unit 1101 is implemented by the communication interface 1402, and the processor 1401 implements the function of the processing unit 1102.

[0244] In another possible implementation, when SBC adopts... Figure 14 When in the form shown, Figure 14 The processor 1401 can call computer execution instructions stored in memory 1403, so that the SBC can execute any of the methods described in any of the above method embodiments, namely SBC1-SBC2.

[0245] This application embodiment does not limit the specific connection medium between the processor 1401 and the memory 1403. In this application embodiment, the memory 1403 and the processor 1401 are connected via a bus 1404, and the bus 1404... Figure 14 The connections between other components are shown in thick lines only and are not intended to be limiting. This bus 1404 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0246] In a simplified embodiment, the terminal device may employ... Figure 15 As shown in the form. (e.g.) Figure 15 The illustrated device 1500 includes at least one processor 1501, a communication interface 1502, and optionally a memory 1503. In one possible implementation, the device 1500 also includes a display screen 1504. The device may also include a sensor 1505, a camera 1506, a microphone 1507, and a speaker 1508. For example, the sensor 1505 may be a touch panel.

[0247] Processor 1501 may have data transmission and reception functions, enabling it to communicate with other devices. For example, in this embodiment, processor 1501 may send media streams to the SBC and receive media streams from the SBC. Figure 15 The device may also include a separate data transceiver module, such as communication interface 1502, for sending and receiving data. When communicating with other devices, processor 1501 can transmit data through communication interface 1502. For example, in this embodiment, processor 1501 can send drawing point data streams, audio data streams, and video data streams to SBC through communication interface 1502, and can also receive drawing point data streams, audio data streams, and video data streams from SBC through communication interface 1502.

[0248] When the terminal device adopts Figure 15 When in the form shown, Figure 15 The processor 1501 can call computer execution instructions stored in the memory 1503, enabling the terminal device to execute the method executed by the terminal device (e.g., the first terminal device-second terminal device) in any of the above method embodiments.

[0249] In one example, memory 1503 stores information for implementing... Figure 12 The computer executes instructions to perform the functions of the communication unit 1202, display unit 1203, processing unit 1201, first acquisition unit 1204, second acquisition unit 1205, and third acquisition unit 1206. Figure 12 The functions / implementation processes of the communication unit 1202, display unit 1203, processing unit 1201, first acquisition unit 1204, second acquisition unit 1205, and third acquisition unit 1206 can all be achieved through... Figure 15 The processor 1501 in the memory calls computer execution instructions stored in memory 1503 to achieve this.

[0250] In another example, memory 1503 stores information for implementing... Figure 12 The computer executes instructions for the function of the display unit 1203. Figure 12 The function / implementation process of the display unit 1203 can be achieved through... Figure 15 The processor 1501 in the memory calls computer execution instructions stored in memory 1503 to achieve this. Figure 12 The function / implementation process of the communication unit can be achieved through Figure 15 It is implemented using the communication interface 1502.

[0251] When the processor 1501 performs the functions of the display unit 1203, if it involves displaying an image, such as displaying a user-drawn pattern, the processor 1501 can display the image through the display screen 1504 in the device 1500. Optionally, when the processor 1501 performs the functions of the display unit, it can also display an image through the display screen of another device, such as sending a display command to another device to instruct it to display an image.

[0252] When executing the function of the first acquisition unit 1204, the processor 1501 can acquire drawing point data through the sensor 1505. When executing the function of the second acquisition unit 1205, the processor 1501 can acquire user audio data through the microphone 1507. The microphone 1507 can also be located outside the device 1500. When executing the function of the third acquisition unit 1206, the processor 1501 can acquire user video data through the camera 1506. The camera 1506 can also be located outside the device 1500.

[0253] In another example, memory 1503 stores information for implementing... Figure 13 The computer executes instructions to perform the functions of the communication unit 1302, display unit 1303, processing unit 1301, and playback unit 1304. Figure 13 The functions / implementation processes of the communication unit 1302, display unit 1303, processing unit 1301, and playback unit 1304 can all be achieved through... Figure 15 The processor 1501 in the memory calls the computer execution instructions stored in the memory 1503 to achieve this.

[0254] When the processor 1501 performs the functions of the display unit 1303, if it involves displaying an image, such as displaying a user-drawn pattern, the processor 1501 can display the image through the display screen 1504 in the device 1500. Optionally, when the processor 1501 performs the functions of the display unit, it can also display an image through the display screen of another device, such as sending a display command to another device to instruct it to display an image.

[0255] When the processor 1501 performs the functions of the playback unit 1304, if the operation involves playing audio data, the processor 1501 can play the user audio data through the speaker 1508 in the device 1500. Optionally, when the processor 1501 performs the function of playing user audio data, it can also play the user audio data through the speaker in another device, such as sending the user audio data to another device.

[0256] This application embodiment does not limit the specific connection medium between the processor 1501, memory 1503, communication interface 1502, display screen 1504, sensor 1505, camera 1506, microphone 1507, and speaker 1508. In this application embodiment, a bus connection is used as an example in the figures, where the bus is represented by a thick line. The connection methods between other components are only for illustrative purposes and are not intended to be limiting. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0257] Based on the above embodiments, this application also provides a computer storage medium storing software programs. When these software programs are read and executed by one or more processors, they can implement the methods executed by the business platform or edge computing device provided in any one or more of the above embodiments. The computer storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0258] Based on the above embodiments, this application also provides a chip, which includes a processor for implementing the functions of a media server or a terminal device involved in any one or more of the above embodiments. Optionally, the chip further includes a memory for storing necessary program instructions and data executed by the processor. This chip can be composed of individual chips or can include chips and other discrete devices.

[0259] It should be understood that the terms "an embodiment," "an implementation," "an embodiment," or "an example" used throughout the specification mean that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, the phrases "in an embodiment," "an implementation," "an embodiment," or "an example" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0260] Furthermore, the terms "system" and "network" are often used interchangeably in this document. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "at least one" in this application refers to one or more, including one, two, three, or more; "multiple" refers to two or more, including two, three, or more. Furthermore, it should be understood that in the description of this application, words such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the terms "comprising" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules and may also include steps or modules not listed.

[0261] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0262] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a network device or a terminal device. Alternatively, the processor and storage medium can exist as discrete components in the network device or terminal device.

[0263] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

Claims

1. A video call method, characterized by, The method comprises: a media server receives a first data stream from a first terminal device, data packets in the first data stream comprising drawing point data of a user-drawn pattern collected by the first terminal device; the media server is deployed in an IP multimedia system (IMS); the media server receives a second data stream from the first terminal device, data packets in the second data stream comprising user audio data and / or user video data of the first terminal device; the media server synchronizes the first data stream and the second data stream according to time parameters of the data packets in the first data stream and time parameters of the data packets in the second data stream to obtain a synchronization processing result; when the data packets in the second data stream comprise user audio data, the media server renders the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user-drawn pattern, and sends the user-drawn pattern and the second data stream to a second terminal device; alternatively, when the data packets in the second data stream comprise user video data, the media server renders the drawing point data in the data packets of the first data stream according to the synchronization processing result to obtain a user-drawn pattern, and renders the user video data in the data packets of the second data stream to obtain a user video image, superimposes the user-drawn pattern on the user video image to obtain a third video stream, and sends the third video stream to the second terminal device; the method further comprises: the media server storing the received data packets belonging to the first data stream in a receiving buffer; rendering the drawing point data in the data packets of the first data stream to obtain a user-drawn pattern comprises: the media server obtains a first data packet belonging to the first data stream from the receiving buffer, and renders the drawing point data in the first data packet as a first drawing point of the user-drawn pattern; when a second data packet belonging to the first data stream is not obtained from the receiving buffer at a rendering rate of the media server for drawing points of the user-drawn pattern, the drawing point data of the second data packet is predicted according to the first N data packets that have been rendered; wherein the sequence number of the second data packet is adjacent to that of the first data packet, and the sequence number of the second data packet is greater than that of the first data packet, or the time stamp of the second data packet is spaced from the time stamp of the first data packet by a time threshold, the time threshold being determined according to a sampling frequency of the first data stream and a sampling frequency of the second data stream; the predicted drawing point data of the second data packet is rendered as a second drawing point of the user-drawn pattern.

2. The method of claim 1, wherein, The rendering rate of the media server for drawing points of the user-drawn pattern is the same as the rate at which the first terminal device collects drawing points of the user-drawn pattern.

3. The method of claim 2, wherein, when the sampling frequency of the first data stream is different from the sampling frequency of the second data stream, the media server synchronizes the first data stream and the second data stream according to time parameters of the data packets in the first data stream and time parameters of the data packets in the second data stream, comprising: When the time parameter comprises a timestamp, the media server synchronizes the data packets of the first data stream and the data packets of the second data stream according to the timestamps of the data packets of the first data stream and a first timestamp increment, the timestamps of the data packets of the second data stream and a second timestamp increment, wherein the first timestamp increment is a difference between timestamps of two adjacent data packets belonging to the first data stream, and the second timestamp increment is a difference between timestamps of two adjacent data packets belonging to the second data stream; or, When the time parameter comprises a sequence number, the media server synchronizes the data packets of the first data stream and the data packets of the second data stream according to the sequence numbers of the data packets of the first data stream, the sequence numbers of the data packets of the second data stream and a sequence number increment, wherein the sequence number increment is determined according to a sampling frequency of the first data stream and a sampling frequency of the second data stream.

4. The method of claim 1, wherein, The method further comprises: When the second data packet is received, replacing the second drawing point with a drawing point rendered according to drawing point data of the received second data packet.

5. The method according to any one of claims 1 to 4, wherein The first data stream is a real-time streaming media transmission protocol (RTP) stream, and the second data stream is an RTP stream; or the first data stream is a real-time streaming media transmission control protocol (RTCP) stream, and the second data stream is an RTCP stream.

6. A video call method, characterized by, The method further comprises: The second terminal device receives a first data stream from the media server, and a data packet in the first data stream comprises drawing point data of a user drawing pattern collected by the first terminal device; The media server is deployed in an IP multimedia system (IMS); The second terminal device receives a second data stream from the media server, and a data packet in the second data stream comprises user audio data and / or user video data of the first terminal device; The second terminal device synchronizes the first data stream and the second data stream according to a time parameter of a data packet in the first data stream and a time parameter of a data packet in the second data stream to obtain a synchronization processing result; When the data packet in the second data stream comprises user audio data, the second terminal device renders the drawing point data of the data packet in the first data stream according to the synchronization processing result to obtain a user drawing pattern, displays the user drawing pattern, and plays the user audio; or, When the data packet in the second data stream comprises user video data, the second terminal device renders the drawing point data in the data packet in the first data stream according to the synchronization processing result to obtain a user drawing pattern, and renders the user video data in the data packet in the second data stream to obtain a user video image, and displays the user drawing pattern superimposed on the user video image; The method further comprises: The second terminal device stores the received data packet belonging to the first data stream in a receiving buffer; The second terminal device renders the drawing point data in the data packet in the first data stream to obtain a user drawing pattern, comprising: The second terminal device obtains a first data packet belonging to the first data stream from the receiving buffer, and renders drawing point data of the first data packet as a first drawing point of a user drawing pattern; when a second data packet belonging to the first data stream is not obtained from the receiving buffer according to a rate at which the second terminal device renders drawing points of the user drawing pattern, drawing point data of the second data packet is predicted according to rendering of the first N data packets belonging to the first data stream that has been performed; wherein a sequence number of the second data packet is adjacent to a sequence number of the first data packet and the sequence number of the second data packet is greater than the sequence number of the first data packet, or an interval time threshold between a time stamp of the second data packet and a time stamp of the first data packet, the interval time threshold being determined according to a sampling frequency of the first data stream and a sampling frequency of the second data stream; rendering the predicted drawing point data of the second data packet as a second drawing point of the user drawing pattern.

7. The method of claim 6, wherein, The rate at which the second terminal device renders drawing points of the user drawing pattern is the same as the rate at which the first terminal device collects drawing points of the user drawing pattern.

8. The method of claim 6, wherein, When the sampling frequency of the first data stream and the sampling frequency of the second data stream are different, the second terminal device synchronizes the first data stream and the second data stream according to time parameters of data packets in the first data stream and time parameters of data packets in the second data stream, including: when the time parameters include time stamps, the second terminal device synchronizes the data packets in the first data stream and the data packets in the second data stream according to time stamps of the data packets in the first data stream and a first time stamp increment, time stamps of the data packets in the second data stream and a second time stamp increment, wherein the first time stamp increment is a difference between time stamps of two adjacent data packets belonging to the first data stream, and the second time stamp increment is a difference between time stamps of two adjacent data packets belonging to the second data stream; or when the time parameters include sequence numbers, the second terminal device synchronizes the data packets in the first data stream and the data packets in the second data stream according to sequence numbers of the data packets in the first data stream, sequence numbers of the data packets in the second data stream, and a sequence number increment, wherein the sequence number increment is determined according to the sampling frequency of the first data stream and the sampling frequency of the second data stream.

9. The method of claim 6, wherein, rendering the predicted second data packet as a second drawing point of the user drawing pattern, the method further comprising: when the second data packet is received, replacing the second drawing point with a drawing point rendered according to drawing point data of the received second data packet.

10. The method according to any one of claims 6 to 9, wherein, The first data stream is a real-time streaming media transmission protocol (RTP) stream, and the second data stream is an RTP stream; or the first data stream is a real-time streaming media transmission control protocol (RTCP) stream, and the second data stream is an RTCP stream.

11. A communications device, characterized by The apparatus includes a communication interface and a processor; The communication interface is configured to transceive signals; The processor is configured to transceive signals through the communication interface and implement the method of any one of claims 1 to 5.

12. A communications device, characterized by The device comprises a communication interface, a processor, a display screen and a player; The communication interface is configured to transceive signals; The display screen is configured to display a user-drawing pattern; The player is configured to play user audio data; The processor is configured to realize the method of any one of claims 6 to 10 by transceiving signals through the communication interface.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when running on a computer, to make the computer execute the method of any one of claims 1 to 10.

Images