Android tablet multi-mode plug flow interaction method and device based on RTC

Abstract

The embodiment of the invention provides an RTC-based Android tablet multi-modal plug flow interaction method and device, and the method comprises the steps: configuring a synchronization timestamp for each piece of modal data in a processor thread when an Android tablet executes RTC plug flow, and achieving the time alignment of each piece of modal data; performing corresponding coding on each piece of modal data after time alignment according to a data type; packaging each piece of coded modal data based on an RTC session protocol, multiplexing an extension head of the current RTC session protocol, and respectively configuring a data type of each piece of modal data in the extension head to form an RTC data packet; and pushing the RTC data packet.

Description

Technical Field

[0001] This invention relates to the field of real-time audio and video communication technology, and in particular to a multimodal streaming interaction method and device for Android tablets based on RTC. Background Technology

[0002] Traditional RTC (Real-Time Communication) streaming solutions on Android tablets only support the transmission of single audio and video streams. They cannot simultaneously integrate the transmission of heterogeneous data from multiple sources, such as touch trajectories, stylus pressure sensitivity, and gravity sensors. This results in a lack of remote interaction information between the communicating parties and poor real-time communication performance. In addition, current multimodal SDKs often run on mobile devices or PCs, lacking a unified abstraction for peripherals such as large tablet screens, external keyboards, and active pens, leading to fragmented interfaces. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of the present invention provide a multimodal streaming interaction method for Android tablets based on RTC, comprising: When an Android tablet performs RTC streaming, a synchronization timestamp is configured for each modal data in the processor thread to achieve time alignment of the data in each modality. Each of the time-aligned modal data is encoded according to its data type. The encoded modal data is encapsulated based on the RTC session protocol, and the extension header of the current RTC session protocol is reused. The data type of each modal data is configured in the extension header to form an RTC data packet. The RTC data packet is pushed.

[0004] In one embodiment, configuring synchronization timestamps for each modal data in the processor thread includes: A unified clock source is used to configure nanosecond-level timestamps for each modal data in the processing thread.

[0005] In one embodiment, pushing the RTC data packet includes: The uplink bandwidth is determined based on the estimated value, and the RTC data packets are pushed based on the determined uplink bandwidth; The method further includes: The protocol extension header based on the RTC data packet determines the data type of each modality involved; Based on the data type, the corresponding decoder is invoked to process the RTC data packet to obtain each modal data; Render the modal data for each of the aforementioned modalities.

[0006] In one embodiment, the method further includes: If transmission link congestion is detected during the push of the RTC data packet, and with the goal of ensuring continuous transmission of the audio stream, packet loss processing is performed on the data in the RTC data packet based on a preset packet loss priority. The packet loss priority includes the order in which the data of each modality in the RTC data packet is processed for packet loss.

[0007] In one embodiment, the packet loss processing based on packet loss priority in the RTC data packet includes: Discard non-keyframe data from the video stream; If the transmission link remains congested after packet loss processing of the video stream data, then packet loss processing is performed on the touch data.

[0008] In one embodiment, the touch data includes touch trajectory and touch handwriting; The packet loss handling for the touch data includes: Based on preserving the frame data corresponding to the starting and ending points of each touch trajectory and touch note, the touch trajectory and touch handwriting are subjected to frequency reduction and thinning processing. Vector interpolation is performed on the touch trajectory and touch handwriting to fill in the missing trajectory points and handwriting points.

[0009] In one embodiment, the method further includes: During the push of the RTC data packets, frame data transmitted within the target time period is cached in a pre-built circular buffer; Listen for the confirmation signal from the data receiving end for each frame of data. In response to receiving the acknowledgement signal, the corresponding frame data is removed from the circular buffer.

[0010] In one embodiment, the method further includes: During the push of the RTC data packet, if it is detected that the data transmission performance of the current transmission link meets the switching conditions, the transmission of the current transmission link is paused, and the last received target acknowledgment signal is recorded. Activate the backup link; The synchronization request sent by the data receiving end is obtained based on the backup link; Based on the synchronization request and target confirmation signal, the unsuccessfully transmitted target frame data stored in the ring buffer is transmitted to the receiving end through the backup link.

[0011] In one embodiment, the method further includes: The receiving end is instructed to render the target frame data transmitted through the backup link at a faster speed, so that the delay of the output rendering result is within a specified time.

[0012] Another embodiment of the present invention also provides an Android tablet multimodal streaming interaction device based on RTC, comprising: The configuration module is used to configure synchronization timestamps for each modal data in the processor thread when the Android tablet performs RTC streaming, so as to achieve time alignment of each modal data. The encoding module is used to encode each of the time-aligned modal data according to its data type. The encapsulation module is used to encapsulate the encoded modal data based on the RTC session protocol, and reuse the extension header of the current RTC session protocol, configuring the data type of each modal data in the extension header to form an RTC data packet. The push module is used to push the RTC data packets.

[0013] Based on the above embodiments, it can be seen that the solution of this application can be applied to Android tablets, and can support multiple peripheral devices to access, realize RTC push of multimodal data between web pages, and the overall process is simple and low in cost.

[0014] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0015] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the multimodal push-stream interaction method for Android tablets based on RTC in an embodiment of the present invention.

[0018] Figure 2 This is a flowchart illustrating a multimodal push-stream interaction method for Android tablets based on RTC, according to another embodiment of the present invention.

[0019] Figure 3 This is a flowchart illustrating the multimodal push-stream interaction method for Android tablets based on RTC in another embodiment of the present invention.

[0020] Figure 4 This is a structural block diagram of the RTC-based multimodal streaming interaction device for Android tablets in an embodiment of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.

[0022] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope of this disclosure will be apparent to those skilled in the art.

[0023] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0024] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0025] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0026] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0027] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.

[0028] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0029] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] To provide a low-latency, multimodal streaming interaction method that can run on / be applied to Android tablets and achieve RTC, such as Figure 1 As shown, this embodiment of the invention provides a multimodal push-stream interaction method for Android tablets based on RTC, including: S1: When an Android tablet performs RTC streaming, a synchronization timestamp is configured for each modal data in the processor thread to achieve time alignment of each modal data. S2: Encode each of the time-aligned modal data according to its data type; S3: Encapsulate the encoded modal data based on the RTC session protocol, reuse the extension header of the current RTC session protocol, and configure the data type of each modal data in the extension header to form an RTC data packet; S4: Push the RTC data packet.

[0031] The multimodal data in this embodiment includes audio data, video data, touch trajectories, touch handwriting (handwriting), and sensor data, such as gravity sensor data. Of course, the multimodal data is not necessarily limited to these; the above data is only the multimodal data in this embodiment and is not intended to limit the types of multimodal data. When the Android tablet executes RTC streaming, a synchronization timestamp is configured for each modal data in the processor thread to achieve time alignment between different modal data based on this timestamp, thereby enabling accurate matching of multimodal data based on time information. After time alignment is completed, the system encodes each modal data according to its data type. For example, audio is encoded in opus48 kbps format, video in H.264 High Profile 720p 30 fps format; touch / handwriting data is compressed (encoded) using Google Protocol Buffers; sensors are encoded using Varint, etc. The specific encoding format is not fixed, but it is preferred to use a structured compact encoding format (protobuf, etc.) for serialization compression. For example, the handwriting data packet contains {x, y, pressure, tilt, timestamp}. Subsequently, the system encapsulates the encoded modal data based on the RTC session protocol and reuses the extension header of the current RTC session, i.e., the extension header of the session protocol. In this extension header, data type information for different modal data is configured in different bit fields, defining the modal type bit field. Similarly, a frame type bit field can be defined. Then, based on the encapsulated multimodal data and the extension header, an RTC data packet is formed. After the RTC data packet is generated, it is pushed to the receiving end, allowing the system's application layer to complete the streaming.

[0032] Based on the above, it can be seen that the solution described in this embodiment can be applied to Android tablets, and can support multiple peripheral devices to access, realize RTC push of multimodal data between web pages, and the overall process is simple and low in cost.

[0033] The step of configuring synchronization timestamps for each modal data in the processor thread includes: S101: Configure nanosecond-level timestamps for each modal data in the processing thread using a unified clock source.

[0034] For example, but not limited to, a unified 90kHz clock source can be used to timestamp 120Hz touch sampling data, 30fps video frame data, 48kHz audio data, and 100Hz sensor data at the nanosecond level, mapping them all to the same time base, thereby achieving frame-level alignment of heterogeneous data. The sequence numbers involved in the marking monotonically increase across modes to facilitate reassembly at the receiver.

[0035] Furthermore, pushing the RTC data packet includes: S401: Determine the uplink bandwidth based on the estimated value, and push the RTC data packet based on the determined uplink bandwidth; For example, the GCC algorithm can be used to estimate uplink bandwidth and determine the estimated bandwidth value. like Figure 2 As shown, the method further includes: S5: The protocol extension header based on the RTC data packet determines the data type of each modality involved; S6: Based on the data type, call the corresponding decoder to process the RTC data packet to obtain each modal data; S7: Render the modal data for each of the aforementioned modalities.

[0036] For example, the receiving end routes the data to the corresponding decoder based on the modal data type identified in the extension header, and then decodes the corresponding modal data based on that decoder. Next, the decoded data is rendered; for example, touch trajectories can be drawn on top of a SurfaceView using OpenGL, and touch strokes can be redrawn in real time using vector paths. This completes the RTC streaming of the multimodal data.

[0037] In one embodiment, the method further includes: S8: During the push of the RTC data packet, if transmission link congestion is detected, with the goal of ensuring continuous transmission of the audio stream, packet loss processing is performed on the data in the RTC data packet based on a preset packet loss priority. The packet loss priority includes the order in which the various modal data in the RTC data packet are processed for packet loss.

[0038] For example, if a transmission link disconnection event is detected, or when combined with the NetworkQuality API (uplink packet loss rate greater than 8% or RTT (Round-Trip Time) greater than 400ms), packet loss is handled according to the packet loss priority. That is, when bandwidth is limited, differentiated packet loss is performed based on modal priority to ensure the correct transmission of audio streams.

[0039] Specifically, such as Figure 3 As shown, the packet loss processing based on packet loss priority in the RTC data packet includes: S801: Discard non-keyframe data from the video stream; S802: If the transmission link is still congested after packet loss processing of the video stream data, then packet loss processing is performed on the touch data.

[0040] For example, first discard B-frame data (non-critical frame data) from the video stream, then observe the link transmission status. If the transmission link is still congested, then perform packet loss processing on the touch data.

[0041] In this embodiment, the touch data includes touch trajectories and touch handwriting. Packet loss processing of the touch data includes: S803: Based on retaining the frame data corresponding to the starting point and ending point of each touch trajectory and touch note, perform frequency reduction and thinning processing on the touch trajectory and touch handwriting; S804: Perform vector interpolation on the touch trajectory and touch handwriting to complete the missing trajectory points and handwriting points.

[0042] For example, the touch sampling frequency can be reduced from 120Hz to 60Hz, while thinning is performed. However, during the thinning process, keyframes representing the "down" and "up" states are forcibly retained, and vector interpolation is performed on the lost trajectory points and handwriting points in the middle to maintain the integrity of the interaction intent under low bandwidth.

[0043] In another embodiment, the method further includes: S9: During the push of the RTC data packet, the frame data transmitted within the target time period is buffered in a pre-built circular buffer; S10: Listen for the confirmation signal for each frame of data received from the data receiver; S11: In response to receiving the acknowledgement signal, remove the corresponding frame data from the circular buffer.

[0044] The scheme described in this embodiment refers to constructing a ring buffer at the sending end to cache in real time the frame data of touch trajectories and touch strokes that were sent out within a specified time period but for which no acknowledgment signal (such as an ACK signal, specifically an ACK mechanism: the receiving end sends back a corresponding ACK_SEQ containing the received frame ID for each frame of key interactive data rendered) from the receiving end (such as the application layer). The specified time period is not limited and can be 3 seconds, 5 seconds, etc. During RTC data packet push, the sending end continuously listens for the acknowledgment signal for each frame of data from the data receiving end. Under normal circumstances, the corresponding frame data in the ring buffer is only cleared after receiving the acknowledgment signal for each sent frame of data.

[0045] Furthermore, the method also includes: S12: During the push of the RTC data packet, if it is detected that the data transmission performance of the current transmission link meets the switching conditions, the transmission of the current transmission link is paused, and the last received target confirmation signal is recorded. S13: Activate backup link; S14: Obtain the synchronization request sent by the data receiving end based on the backup link; S15: Based on the synchronization request and target confirmation signal, the unsuccessfully transmitted target frame data stored in the ring buffer is transmitted to the receiving end through the backup link.

[0046] The method further includes: S16: Instruct the receiving end to render the target frame data transmitted through the backup link at double speed, so that the delay of the output rendering result is within the specified time.

[0047] The above embodiment refers to the following: during the push of the RTC data packet, if severe congestion is detected on the transmission link, or if the packet loss rate and RTT both exceed preset thresholds, it can be determined that the current performance of the transmission link is poor and meets the conditions for link switching. At this time, the system will suspend data transmission on the current transmission link, that is, deactivate the link, and build a backup link between the sending and receiving ends to maintain the connection of the DataChannel between the sending and receiving ends. For example, the backup RTC configuration can be obtained through a signaling server, and the backup link can be built by hot-plugging a new Candidate using setRemoteDescription. At the same time, the sequence number Last_Ack_Seq of the last successfully received ACK on the deactivated transmission link is recorded, and this sequence number matches the frame ID of the corresponding frame data. In application, this can be achieved, for example, by calling createOffer to generate a snapshot SDP and saving the DataChannel state. When the establishment of the new channel is completed, that is, when the handshake authentication is completed, the latest completed frame ID sent by the receiving end is obtained through the backup link. At this point, the system identifies the failed-to-transmit frame data from the circular buffer area based on the frame ID and recorded information, and retransmits the frame data via the backup link. The transmission speed can be increased based on the actual situation, i.e., accelerated transmission. Simultaneously, the system can instruct the receiving end, or the receiving end can determine from the pre-negotiated content that the frame data retransmitted via the backup link needs to be rendered at a faster speed, such as 4x speed, to quickly render and redraw, completing missing trajectories or handwriting. This ensures that the perceived "lag" is compressed within a specified time period, such as within 200ms, while guaranteeing that trajectory points and handwriting points are not lost or out of order.

[0048] like Figure 4 As shown, another embodiment of the present invention also provides an Android tablet multimodal streaming interaction device based on RTC, comprising: The configuration module is used to configure synchronization timestamps for each modal data in the processor thread when the Android tablet performs RTC streaming, so as to achieve time alignment of each modal data. The encoding module is used to encode each of the time-aligned modal data according to its data type. The encapsulation module is used to encapsulate the encoded modal data based on the RTC session protocol, and reuse the extension header of the current RTC session protocol, configuring the data type of each modal data in the extension header to form an RTC data packet. The push module is used to push the RTC data packets.

[0049] In one embodiment, configuring synchronization timestamps for each modal data in the processor thread includes: A unified clock source is used to configure nanosecond-level timestamps for each modal data in the processing thread.

[0050] In one embodiment, pushing the RTC data packet includes: The uplink bandwidth is determined based on the estimated value, and the RTC data packets are pushed based on the determined uplink bandwidth; The device further includes: The determination module is used to determine the data type of each modal data involved based on the protocol extension header of the RTC data packet; The decoding module is used to call the corresponding decoder based on the data type to process the RTC data packet and obtain each modal data; The rendering module is used to render the various modal data.

[0051] In one embodiment, the device further includes: The packet loss processing module is used to process the data in the RTC data packet based on a preset packet loss priority, with the goal of ensuring the continuous transmission of the audio stream, if transmission link congestion is detected during the push of the RTC data packet. The packet loss priority includes the order in which the various modal data in the RTC data packet are processed for packet loss.

[0052] In one embodiment, the packet loss processing based on packet loss priority in the RTC data packet includes: Discard non-keyframe data from the video stream; If the transmission link remains congested after packet loss processing of the video stream data, then packet loss processing is performed on the touch data.

[0053] In one embodiment, the touch data includes touch trajectory and touch handwriting; The packet loss handling for the touch data includes: Based on preserving the frame data corresponding to the starting and ending points of each touch trajectory and touch note, the touch trajectory and touch handwriting are subjected to frequency reduction and thinning processing. Vector interpolation is performed on the touch trajectory and touch handwriting to fill in the missing trajectory points and handwriting points.

[0054] In one embodiment, the device further includes: A caching module is used to cache frame data transmitted within a target time period in a pre-built circular buffer during the push of the RTC data packets; The monitoring module is used to monitor the acknowledgment signal for each frame of data returned by the data receiving end. A removal module is used to remove the corresponding frame data from the circular buffer in response to receiving the acknowledgement signal.

[0055] In one embodiment, the device further includes: The detection module is used to pause the transmission of the current transmission link and record the last received target acknowledgment signal when the data transmission performance of the current transmission link meets the switching conditions during the push of the RTC data packet. The startup module is used to start the backup link; The acquisition module is used to acquire the synchronization request sent by the data receiving end based on the backup link; The retransmission module is used to transmit the unsuccessfully transmitted target frame data stored in the ring buffer to the receiving end through the backup link based on the synchronization request and the target confirmation signal.

[0056] In one embodiment, the device further includes: The instruction module is used to instruct the receiving end to render the target frame data transmitted through the backup link at double speed, so that the delay of the output rendering result is within a specified time.

[0057] Another embodiment of the present invention also provides an electronic device, comprising: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the RTC-based multimodal push-stream interaction method for Android tablets as described in any of the above descriptions.

[0058] Furthermore, one embodiment of the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the RTC-based multimodal streaming interaction method for Android tablets as described above. It should be understood that the various solutions in this embodiment have the corresponding technical effects in the above-described method embodiments, and will not be repeated here.

[0059] Furthermore, embodiments of the present invention also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions, which, when executed, cause at least one processor to perform an RTC-based multimodal push-stream interaction method for Android tablets, such as the embodiment described above.

[0060] It should be noted that the computer storage medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.

[0061] Furthermore, those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0062] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

Claims

1. A multimodal push-stream interaction method for Android tablets based on RTC, characterized in that, include: When an Android tablet performs RTC streaming, a synchronization timestamp is configured for each modal data in the processor thread to achieve time alignment of the data in each modality. Each of the time-aligned modal data is encoded according to its data type. The encoded modal data is encapsulated based on the RTC session protocol, and the extension header of the current RTC session protocol is reused. The data type of each modal data is configured in the extension header to form an RTC data packet. The RTC data packet is pushed.

2. The multimodal streaming interaction method for Android tablets based on RTC according to claim 1, characterized in that, The configuration of synchronization timestamps for each modal data in the processor thread includes: A unified clock source is used to configure nanosecond-level timestamps for each modal data in the processing thread.

3. The multimodal streaming interaction method for Android tablets based on RTC according to claim 1, characterized in that, The pushing of the RTC data packet includes: The uplink bandwidth is determined based on the estimated value, and the RTC data packets are pushed based on the determined uplink bandwidth; The method further includes: The protocol extension header based on the RTC data packet determines the data type of each modality involved; Based on the data type, the corresponding decoder is invoked to process the RTC data packet to obtain each modal data; Render the modal data for each of the aforementioned modalities.

4. The multimodal streaming interaction method for Android tablets based on RTC according to claim 1, characterized in that, The method further includes: If transmission link congestion is detected during the push of the RTC data packet, and with the goal of ensuring continuous transmission of the audio stream, packet loss processing is performed on the data in the RTC data packet based on a preset packet loss priority. The packet loss priority includes the order in which the data of each modality in the RTC data packet is processed for packet loss.

5. The multimodal push-streaming interaction method for Android tablets based on RTC according to claim 4, characterized in that, The packet loss processing based on packet loss priority in the RTC data packets includes: Discard non-keyframe data from the video stream; If the transmission link remains congested after packet loss processing of the video stream data, then packet loss processing is performed on the touch data.

6. The multimodal streaming interaction method for Android tablets based on RTC according to claim 5, characterized in that, The touch data includes touch trajectories and touch handwriting; The packet loss handling for the touch data includes: Based on preserving the frame data corresponding to the starting and ending points of each touch trajectory and touch note, the touch trajectory and touch handwriting are subjected to frequency reduction and thinning processing. Vector interpolation is performed on the touch trajectory and touch handwriting to fill in the missing trajectory points and handwriting points.

7. The multimodal streaming interaction method for Android tablets based on RTC according to claim 1, characterized in that, The method further includes: During the push of the RTC data packets, frame data transmitted within the target time period is cached in a pre-built circular buffer; Listen for the confirmation signal from the data receiving end for each frame of data. In response to receiving the acknowledgement signal, the corresponding frame data is removed from the circular buffer.

8. The multimodal streaming interaction method for Android tablets based on RTC according to claim 7, characterized in that, The method further includes: During the push of the RTC data packet, if it is detected that the data transmission performance of the current transmission link meets the switching conditions, the transmission of the current transmission link is paused, and the last received target acknowledgment signal is recorded. Activate the backup link; The synchronization request sent by the data receiving end is obtained based on the backup link; Based on the synchronization request and target confirmation signal, the unsuccessfully transmitted target frame data stored in the ring buffer is transmitted to the receiving end through the backup link.

9. The multimodal streaming interaction method for Android tablets based on RTC according to claim 8, characterized in that, The method further includes: The receiving end is instructed to render the target frame data transmitted through the backup link at a faster speed, so that the delay of the output rendering result is within a specified time.

10. A multimodal streaming interaction device for Android tablets based on RTC, characterized in that, include: The configuration module is used to configure synchronization timestamps for each modal data in the processor thread when the Android tablet performs RTC streaming, so as to achieve time alignment of each modal data. The encoding module is used to encode each of the time-aligned modal data according to its data type. The encapsulation module is used to encapsulate the encoded modal data based on the RTC session protocol, and reuse the extension header of the current RTC session protocol, configuring the data type of each modal data in the extension header to form an RTC data packet. The push module is used to push the RTC data packets.

Images