A multi-modal data transmission method, apparatus, medium and electronic device

Abstract

The specification provides a multi-modal data transmission method, device, medium and electronic equipment. In the method, the sending end of the multi-modal interaction system can first determine a to-be-tested maximum transmission unit (MTU) value, and then sends a probe packet configured to disallow fragmentation to a receiving end as a receiving object to probe the first link from the sending end to a media service node and the second link from the media service node to the receiving end. In the process, if local link state abnormal information returned by an intermediate network device included in the first link or the second link is received, it is determined that the to-be-tested MTU value is an unusable MTU value. Conversely, if the confirmation information returned by the receiving end is received, it is determined that the to-be-tested MTU value is a usable MTU value. Finally, according to each usable MTU value, a maximum non-fragmentation MTU value commonly supported by all links in the entire transmission path from the sending end to the receiving end via the media service node can be determined as a target MTU value.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of communication technology, and in particular to a multimodal data transmission method, apparatus, medium and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence and communication technologies, multimodal real-time interaction has become an important application scenario, covering areas such as intelligent agent AI audio and video interaction, cross-terminal video conferencing, real-time live streaming, and cloud gaming. These scenarios place extremely high demands on data transmission latency, stability, and throughput. To meet the need for low-latency transmission, the User Datagram Protocol (UDP) is widely used in multimodal data transmission due to its connectionless nature, low overhead, and the fact that it does not require a handshake mechanism.

[0003] When transmitting data using the UDP protocol, the Maximum Transmission Unit (MTU) is a critical network parameter that defines the maximum data length that a data link layer frame can carry. However, the UDP protocol itself lacks a built-in path MTU discovery mechanism. Therefore, it typically relies on a preset MTU value from the sender or a simple probing mechanism to determine the MTU value. The accuracy of the MTU value determined in this way is poor, and it is very easy to encounter situations where the set MTU value results in low bandwidth utilization or even unusable bandwidth. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a multimodal data transmission method is proposed, the method being applied to a transmitting end of a multimodal interaction system, the multimodal interaction system comprising: the transmitting end, a media service node, and a receiving end, wherein the transmitting end communicates with the media service node via a first link, and the receiving end communicates with the media service node via a second link, the method comprising: Determine the maximum transmission unit (MTU) value to be tested, and create a probe packet configured to not allow fragmentation according to the MTU value to be tested; The probe packet is sent to the receiving end to probe the first link and the second link; If the local link status abnormality information of the first link is received from the intermediate network device included in the first link, or the local link status abnormality information of the second link is received from the media service node, then the MTU value to be tested is determined to be an unusable MTU value. If an acknowledgment is received from the receiving end, the MTU value to be tested is determined to be an available MTU value. Based on the available MTU value, a target MTU value is selected; and service data is sent based on the target MTU value.

[0005] According to a second aspect of one or more embodiments of this specification, a multimodal data transmission method is provided, the method being applied to a media service node of a multimodal interaction system, the multimodal interaction system comprising: a sending end, the media service node, and a receiving end, wherein the sending end communicates with the media service node via a first link, and the receiving end communicates with the media service node via a second link, the method comprising: Receive probe packets sent by the sending end and intended for the receiving end, wherein the probe packets are created according to the maximum transmission unit (MTU) value to be tested and are configured to not fragment. The probe packet is sent to the receiving end to probe the second link; If an Internet Control Message Protocol (ICMP) message is received from an intermediate network device included in the second link, the ICMP message is parsed to generate local link state anomaly information for the second link and returned to the sending end, so that the sending end can determine the MTU value to be tested as an unusable MTU value based on the local link state anomaly information. If the receiving end receives the confirmation information, it forwards the confirmation information to the sending end so that the sending end can determine the MTU value to be tested as an available MTU value based on the confirmation information.

[0006] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the multimodal data transmission method described above by executing the executable instructions.

[0007] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the multimodal data transmission method described above.

[0008] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the multimodal data transmission method described above.

[0009] In this method, the transmitter of the multimodal interaction system first determines the maximum transmission unit (MTU) value to be tested, and then sends probe packets configured to disallow fragmentation to the receiver to probe the first link from the transmitter to the media service node and the second link from the media service node to the receiver. During this process, if a partial link status anomaly information is received from an intermediate network device of the first link, or a partial link status anomaly information is received from the media service node of the second link, the MTU value to be tested is determined to be an unusable MTU value. Conversely, if an acknowledgment information is received from the receiver, the MTU value to be tested is determined to be an usable MTU value. Finally, based on each usable MTU value, the target MTU value is determined, and multimodal service data is sent based on the target MTU value.

[0010] As can be seen from the above method, the sending end can send probe packets configured to disallow fragmentation and perform global link probing on the first link from the sending end to the media service node and the second link from the media service node to the receiving end. Specifically, when receiving local link status anomaly information returned by the intermediate network device of the first link, or receiving local link status anomaly information returned by the media service node of the second link, it can be determined that the current MTU value to be tested exceeds the carrying capacity of the link, and thus it is marked as an unusable MTU value. Conversely, if receiving confirmation information returned by the receiving end, it is marked as an usable MTU value. Finally, based on the usable MTU value, the maximum unfragmented MTU value supported by all links in the entire transmission path from the sending end through the media service node to the receiving end can be determined as the target MTU value. This can avoid problems such as data packets being forcibly fragmented at intermediate network devices due to an excessively large MTU value, which leads to fragmentation loss, or insufficient bandwidth utilization due to an excessively small MTU value. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a multimodal interaction system provided in an exemplary embodiment.

[0012] Figure 2 This is a flowchart illustrating a multimodal data transmission method provided in an exemplary embodiment.

[0013] Figure 3 This is a schematic diagram of a detection process provided in an exemplary embodiment.

[0014] Figure 4 This is a schematic diagram of the process for determining the target MTU value provided in an exemplary embodiment.

[0015] Figure 5 This is a flowchart illustrating a multimodal data transmission method provided in an exemplary embodiment.

[0016] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment.

[0017] Figure 7 This is a block diagram of a multimodal data transmission device provided in an exemplary embodiment.

[0018] Figure 8 This is a block diagram of a multimodal data transmission device provided in an exemplary embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0020] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0021] In current multimodal real-time interaction scenarios, such as AI audio and video interaction with intelligent agents, cross-terminal video conferencing, and cloud gaming, the User Datagram Protocol (UDP) is commonly used to achieve low-latency data transmission. However, the UDP protocol itself lacks a built-in Maximum Transmission Unit (MTU) discovery mechanism. If the MTU value set by the sender is too large, data packets will be forced to undergo IP layer fragmentation when passing through intermediate network devices (such as routers and firewalls). In mobile network (4G / 5G / Wi-Fi) environments, the probability of fragment loss is high, which can easily lead to audio and video stuttering, screen tearing, or delays in interactive commands. Conversely, if the MTU value is set too small, although fragmentation is avoided, it will lead to low bandwidth utilization, increased data transmission header overhead, and thus affect the throughput and audio and video quality of multimodal data.

[0022] Furthermore, in multimodal real-time interactive scenarios, such as multi-person video conferencing, online live streaming, and distance education, a single sender often needs to simultaneously transmit audio and video data to multiple receivers. In such cases, if a traditional point-to-point (P2P) direct connection is used, the uplink bandwidth pressure on the sender will increase linearly with the number of receivers, and it will be difficult to adapt to the diverse network environments of different receivers. To reduce end-to-end transmission latency and support large-scale concurrent user access, a multi-hop global link architecture of "sender-media service node-receiver" is typically adopted. This architecture uses the media service node as a relay hub to decouple the sender and receiver connections.

[0023] Specifically, media service nodes can establish independent transmission connections (such as UDPSockets) with both the sender and receiver. For the sender, the media service node is its direct communication peer; for the receiver, the media service node is the source of the data. Based on this, the media service node can receive audio and video streams uploaded by the sender and, according to the receiver's subscription requirements, network bandwidth, and protocol compatibility, selectively forward, adjust the bitrate, or encapsulate the data stream without performing time-consuming decoding and re-encoding of the media content, thus achieving efficient distribution while ensuring low latency.

[0024] However, this decoupling approach also leads to new problems: because the media service node terminates the original transmission connection at the sending end, it logically splits the end-to-end physical path into two independent links: a first link (from the sending end to the media service node) and a second link (from the media service node to the receiving end). These two links may span completely different network domains (such as public networks, intranets, 4G / 5G mobile networks, etc.), and their maximum MTU limits are often different and invisible to each other. Therefore, if the MTU value is determined based solely on local link information or if the global link MTU is assumed to be consistent, the determined MTU value can easily exceed or fall far below the actual carrying capacity of a certain link segment, causing problems such as data packet fragmentation, loss, or low bandwidth utilization, which in turn seriously affects the stability of multimodal interaction.

[0025] Based on this, this specification provides a multimodal data transmission method. This method is applied to the sending end of a multimodal interactive system. By sending probe packets configured to disallow fragmentation, global link probing is performed on the first link from the sending end to the media service node and the second link from the media service node to the receiving end. When a local link status anomaly information is received from an intermediate network device in either the first or second link, it can be determined that the current MTU value to be tested exceeds the carrying capacity of that link, thus marking it as an unusable MTU value. Conversely, if an acknowledgment information is received from the receiving end, it is marked as an usable MTU value. Finally, based on the usable MTU value, the maximum unfragmented MTU value supported by all links in the entire transmission path from the sending end through the media service node to the receiving end can be determined as the target MTU value. This avoids problems such as data packet loss caused by forced fragmentation at intermediate network devices due to an excessively large MTU value, or insufficient bandwidth utilization due to an excessively small MTU value.

[0026] In order to clearly describe the technical solution provided in this application, some core concepts involved in this application will be explained below.

[0027] Multimodal real-time interaction refers to real-time interaction scenarios based on various data formats such as audio / video, voice commands, gestures, and text / images. These scenarios have extremely high requirements for transmission latency, stability, and throughput. Typical applications include multimodal AI agent dialogue, AR / VR collaborative work, real-time audio / video calls, and cloud gaming. To reduce transmission latency, multimodal data is typically transmitted via the UDP protocol.

[0028] MTU (Maximum Transmission Unit): This refers to the maximum length of the data portion that a data link layer frame can carry, excluding the frame header. If an upper-layer data packet exceeds this size, the IP layer must fragment it for transmission. The core objective of this method is to determine the maximum MTU value that an end-to-end global link can support without triggering IP layer fragmentation.

[0029] Media service node (e.g., Selective Forwarding Unit, SFU): A server device deployed at the network edge or in the cloud, specifically designed for relaying and distributing multimodal data streams between multiple terminals. In a multimodal interactive system, the media service node acts as the core hub for global link communication, logically decoupling the sending end from one or more receiving ends by establishing independent transmission connections.

[0030] DF bit (Don't Fragment): A flag bit in the IP header. When this bit is set, it instructs intermediate network devices not to fragment the packet. If the packet size exceeds the link MTU, the network device will discard the packet and return an Internet Control Message Protocol (ICMP) message, which can be an error message indicating "fragmentation is required but the DF bit is set".

[0031] Raw Socket: A programming interface that allows applications to directly access network layer or transport layer underlying protocol data. Unlike ordinary sockets, when using Raw Socket, developers can manually construct and parse the complete packet structure, including the IP header and ICMP header.

[0032] Local link status anomaly information: This refers to feedback information generated when a probe packet is transmitted on a local link according to the MTU value to be tested, and the MTU value to be tested exceeds the carrying capacity of the local link, and the probe packet is configured not to be fragmented, thus indicating that there is an MTU anomaly in the local link. In the first link, the local link status anomaly information can be anomaly information carried in the Internet Control Message Protocol (ICMP) messages returned by intermediate network devices included in the first link, or anomaly information carried in the transport layer status feedback event generated by the operating system based on the ICMP messages. In the second link, the local link status anomaly information can be anomaly information carried in the ICMP messages returned by intermediate network devices included in the second link to the media service node, or application layer anomaly feedback information generated by the media service node after parsing the ICMP messages and fed back to the sending end.

[0033] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic diagram of a multimodal interaction system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a transmitter 11, a media service node 12, a receiver 13, and a network 14.

[0035] The sending end 11 and receiving end 13 can be physical devices containing an independent host, or virtual devices hosted by a host cluster. During operation, the sending end 11 can run client-side or server-side programs of a multimodal interactive application to generate and send multimodal service data; the receiving end 13 can run corresponding receiving-side programs to receive and process multimodal service data. The sending end 11 and receiving end 13 are only some types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as personal computers (PCs), tablets, laptops, PDAs, wearable devices (such as smart glasses, smartwatches, etc.), smartphones, etc., and one or more embodiments in this specification do not limit this. The client-side program of the aforementioned multimodal interactive application can be a native application installed on the electronic device, or the client-side program can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be implemented through a page displayed by a browser. Here, the browser can be a standalone browser application or a browser module embedded in some applications.

[0036] Media service node 12 is used to relay and distribute data between sender 11 and receiver 13. Media service node 12 can be a Selective Forwarding Unit (SFU), which can be a physical server deployed on an edge computing node or a virtualized service instance. During operation, media service node 12 runs the corresponding forwarding service program, responsible for forwarding multimodal data streams from sender 11 to receiver 13, or distributing them to multiple receivers in many-to-many interaction scenarios.

[0037] For the network 14 that facilitates interaction between the sending end 11, the media service node 12, and the receiving end 13, communication can be implemented using either wired or wireless networks, depending on the communication methods supported by the respective devices. This specification does not impose any restrictions on this. For example, if the sending end 11 is a PC, it can use a wired Ethernet or wireless Wi-Fi network for communication as needed; if the sending end 11 is a mobile phone, it typically uses a mobile communication network (such as 4G or 5G) or a wireless local area network. The network 14 may include intermediate network devices such as routers, switches, and firewalls, which constitute the global link transmission path from the sending end to the receiving end.

[0038] Figure 2 This is a flowchart illustrating a multimodal data transmission method provided in an exemplary embodiment, including: S200: Determine the maximum transmission unit (MTU) value to be tested, and create a probe packet configured to not allow fragmentation according to the MTU value to be tested.

[0039] S202: Send the probe packet to the receiving end to probe the first link and the second link.

[0040] In this specification, the sending end of the multimodal interaction system can send multimodal data (such as audio, video, text, etc.) to the receiving end according to business needs. Before that, it is necessary to determine the fragmentation-free MTU value of the global link to ensure that the data packets can be transmitted smoothly on the network path without being fragmented by intermediate devices.

[0041] At this point, the sending end can first determine at least one MTU value to be tested, and send probe packets to the receiving end according to the MTU value to be tested, so as to probe the first link from the sending end to the media service node and the second link from the media service node to the receiving end.

[0042] The MTU value to be tested can be randomly generated or determined based on historical network link MTU distribution statistics and protocol standards.

[0043] The first and second links described above can consist of multiple intermediate network devices, such as routers, firewalls, and Network Address Translation (NAT) devices. This specification does not impose any restrictions on the type, number, or configuration of intermediate networks. In this specification, the naming of the first and second links is for distinction only and does not imply a symmetrical relationship between them.

[0044] Furthermore, in this specification, the sending end can first send a probe packet to complete the full detection process of the global link MTU, and after determining the target MTU value, then perform multimodal service data transmission according to the target MTU value.

[0045] However, if a probe packet is sent first to complete the full probe process of the global link MTU before multimodal service data transmission, it may introduce additional startup delay, which will increase the perceived latency of the first packet and affect the interactive experience.

[0046] Therefore, the sending end can determine an initial MTU value (such as a preset minimum safe MTU value or an MTU value determined based on historical network link MTU distribution statistics) during the initialization phase for use in sending multimodal service data. Then, during the detection of the target MTU value, the sending end can simultaneously send service data packets corresponding to the multimodal service data according to this initial MTU value, achieving parallel execution of MTU detection and service data transmission. This reduces transmission latency while ensuring reliability. Specifically, for example... Figure 3 As shown.

[0047] Figure 3This is a schematic diagram of a detection process provided in an exemplary embodiment.

[0048] Combination Figure 3 It can be seen that the sending end can first execute the initialization task. During the initialization phase, the initial MTU value is detected and discovered. Then, the multimodal service data can be packetized and sent according to the initial MTU value. At the same time, at least one MTU value to be tested can be determined, and the global link target MTU value can be probed based on the determined MTU value to be tested.

[0049] Specifically, the sending end can construct a probe packet with a length equal to the determined MTU value to be tested, and then send the probe packet to the receiving end to probe the first link from the sending end to the media service node and the second link from the media service node to the receiving end.

[0050] It should be noted that in practical applications, the sending end can use a Raw Socket to construct and send probe packets with the DF flag, and use the Raw Socket to capture ICMP messages returned in the first or second link, and then obtain the result of the target MTU value probe based on the ICMP messages.

[0051] Specifically, the sending end can construct the IP header in the probe packet using a Raw Socket, set the DF bit in the IP header to 1, and then send a probe packet of the same length as the MTU value to be tested. If the actual MTU limit (also known as the egress MTU) of a node in the first or second link is less than the length of the probe packet, the node will discard the probe packet and return an ICMP message (such as a "Fragmentation Needed" message of type 3 code 4 in IPv4).

[0052] Based on this, the sending end can parse the ICMP message to know the detection results of the first or second link, and then determine the target MTU value based on the detection results.

[0053] However, in practical applications, directly using Raw Sockets to capture ICMP packets to obtain probe results is usually feasible on PCs, but it has the following problems in mobile operating systems: In mobile operating systems, creating a Raw Socket often requires root or highest administrator privileges. Granting root privileges to ordinary applications can compromise the system's security sandbox mechanism, leading to serious security risks. Therefore, mobile operating systems typically prohibit ordinary applications from obtaining root privileges. Consequently, mobile operating systems may be unable to directly capture underlying ICMP control packets based on Raw Sockets, making it impossible to probe the target MTU value.

[0054] Therefore, to address the aforementioned issues, this specification allows the sending end to pre-create a service socket and a listening socket during the initialization phase. The service socket can be used to actually send probe packets and multimodal service data, and can also be used to receive local link state anomaly information returned by the media service node. The listening socket can be used to capture local link state anomaly information returned by the first link. Specifically, on a PC (e.g., Windows, Linux, macOS), this listening socket can capture ICMP messages returned by intermediate network devices included in the first link via Raw Socket, thereby parsing and extracting the local link state anomaly information. On a mobile device (e.g., Android, iOS), if the mobile operating system grants access permissions to underlying network data, the sending end has the permission to directly read the raw data packets of the protocol stack. In this case, the sending end can intercept network layer IP packets through the listening socket and parse the header and payload of the captured IP packets to extract the local link state anomaly information.

[0055] Furthermore, if the mobile operating system restricts access to underlying network data, the sending end is prohibited from directly intercepting network layer IP packets. However, when the operating system kernel receives ICMP messages returned by intermediate network devices, it automatically converts them into transport layer status feedback events (such as error states or callback events). In this case, the standard network status listening interface provided by the operating system can be called through the listening socket to listen for transport layer status feedback events, thereby obtaining local link status anomaly information without obtaining root privileges.

[0056] It should be noted that, for the second link, the ICMP messages generated by the intermediate network devices in the second link first reach the media service node. Since the media service node is deployed on the server side and has complete network protocol stack parsing capabilities, when it receives an ICMP timeout message from the downstream link, it can strip off the network layer header and convert it into application layer feedback signaling, which is then transmitted back to the sender via UDP. At this point, because this feedback signaling has been encapsulated as a standard service payload, the sender can directly receive the feedback signaling through a service socket and parse the link state identifier within it to obtain local link state anomaly information for the second link.

[0057] The aforementioned feedback signaling refers to the application layer control data packet reconstructed by the media service node based on the received ICMP messages. Specifically, after capturing the ICMP messages fed back by intermediate network devices in the second link, the media service node can parse and extract data such as the ICMP type and the original UDP 5-tuple characteristics, and then perform structured encapsulation according to a predefined encapsulation format to generate the application layer control data packet, i.e., the feedback signaling. The aforementioned predefined encapsulation format can be set according to actual needs, for example: a fixed-length structure or a Type-Length-Value (TLV).

[0058] Building upon this, a Full-Link MTU Detection (FMD) module can be deployed on the sending end. This module can operate as an independent component of the real-time communication engine. The FMD module includes an interface layer, a core control module (FMD-core), a UDP socket module (FMD-udp socket), and a raw socket module (FMD-raw socket). Figure 3 As shown, the FMD module achieves full-process management from initialization to dynamic detection through the collaborative work of its various sub-modules.

[0059] During the initialization phase, the FMD interface layer can receive trigger commands from the sender to start the FMD component. After that, the FMD-core module can perform initialization configuration, including setting the initial MTU value, and can instruct the UDP socket module to create a service socket and instruct the raw socket module to create a listening socket.

[0060] In addition, due to differences in mobile operating system version iterations, vendor customization strategies, or underlying network protocol stack implementations, some mobile systems may still be unable to listen for transport layer status feedback events through the standard network status listening interface to obtain local link status anomaly information for the first link. For these mobile systems, this specification can also detect the target MTU value based solely on the local link status anomaly information returned by intermediate network devices included in the second link and the acknowledgment information returned by the receiving end, which will be explained in detail later.

[0061] Furthermore, after obtaining the local link status anomaly information returned by the intermediate network device included in the first link or the second link, the sending end can determine the target MTU value, as shown in steps S204 to S208.

[0062] S204: If the local link status abnormality information of the first link is received from the intermediate network device included in the first link, or the local link status abnormality information of the second link is received from the media service node, then the MTU value to be tested is determined to be an unusable MTU value.

[0063] S206: If an acknowledgment is received from the receiving end, the MTU value to be tested is determined to be an available MTU value.

[0064] When the sending end receives local link status abnormality information returned by the intermediate network device included in the first link or the second link, it can determine that the MTU value to be tested currently used exceeds the actual MTU limit of a certain node in the first link or the second link. At this time, the MTU value to be tested can be marked as an unusable MTU value.

[0065] Conversely, if the sending end does not receive any local link status anomaly information from the intermediate network devices included in the first or second link after sending the probe packet, but instead receives an acknowledgment from the receiving end, it indicates that the probe packet of that size successfully passed through the first and second links without being dropped or fragmented at intermediate nodes due to MTU limitations. In this case, the sending end determines that the MTU value to be tested is an available MTU value.

[0066] In addition, regarding the situation mentioned above where only the local link status anomaly information returned by the intermediate network devices included in the second link can be obtained, the sending end can determine that the MTU value to be tested is an unusable MTU value after obtaining the local link status anomaly information returned by the intermediate network devices included in the second link; after receiving the acknowledgment information returned by the receiving end, determine that the MTU value to be tested is an usable MTU value; and, within a specified time period of sending the probe packet (the specified time period can be set according to actual needs), if no acknowledgment information returned by the receiving end is received, and no local link status anomaly information returned by the intermediate network devices included in the second link is obtained, determine that the MTU value to be tested is an unusable MTU value.

[0067] In this specification, the acknowledgment information returned by the receiving end may refer to the reachability acknowledgment at the transport layer or the ACK response information returned by the application layer specifically for probe packets. This specification does not impose any restrictions on this.

[0068] S208: Select a target MTU value based on the available MTU value; and send service data based on the target MTU value.

[0069] Furthermore, after determining the detection results (i.e., usable or unusable) of the aforementioned MTU value to be tested, the sending end can use different strategies to determine the final target MTU value based on the number of MTU values ​​to be tested and the detection results. Specifically, this can be divided into two cases: one MTU value to be tested and multiple MTU values ​​to be tested. These two cases will be explained in detail below.

[0070] If the sender determines that the MTU value to be tested is an available MTU value, it indicates that the current global link may support a larger transmission unit. In this case, the sender can adjust the MTU value to be tested to obtain an adjusted MTU value, and then re-probe based on the adjusted MTU value until the detected adjusted MTU value is determined to be an unavailable MTU value or the preset maximum number of probes threshold is reached. At this point, the last MTU value confirmed as available can be used as the target MTU value. The adjusted MTU value to be tested is larger than the MTU value to be tested.

[0071] Furthermore, if the sending end determines that the MTU value to be tested is an unavailable MTU value, it indicates that the MTU value to be tested exceeds the global link capacity. In this case, the sending end can adjust the MTU value to obtain an adjusted MTU value, and then re-probe based on the adjusted MTU value until the detected MTU value is determined to be an available MTU value or reaches a preset minimum threshold. At this point, the MTU value initially confirmed as available can be used as the target MTU value; wherein the adjusted MTU value is smaller than the original MTU value.

[0072] When multiple MTU values ​​are identified for testing, the sending end selects the smallest value from the remaining MTU values ​​as the current MTU value and begins probing based on this value. If the current MTU value is determined to be usable, the sending end selects the largest value from the remaining MTU values ​​as the new current MTU value and begins probing again based on this new value. This process continues until all MTU values ​​have been traversed or the first unusable MTU value is encountered. At this point, the last usable MTU value is selected as the target MTU value. Alternatively, the sending end can select the largest value from the remaining MTU values ​​as the current MTU value and proceed with sequential probing in descending order. In this case, once a usable MTU value is detected, it can be directly identified as the target MTU value without needing to probe for smaller MTU values. This method will not be elaborated upon further in this specification.

[0073] It should be noted that because the probe packets sent during the MTU detection process compete for bandwidth resources with the service data packets corresponding to the multimodal service data sent in parallel, this can lead to increased transmission delays, audio and video stuttering, or a degraded interactive experience for the multimodal service data. Therefore, the more efficient and shorter the detection process used to determine the target MTU value, the less impact it will have on normal service transmission and the better the user experience.

[0074] Therefore, in order to minimize the interference of the detection process on service transmission while ensuring detection accuracy, in this specification, the sending end can also generate each detection packet according to each MTU value when there are multiple MTU values ​​to be tested, and then send each detection packet in parallel to the receiving end as the receiving object, so as to detect the first link from the sending end to the media service node and the second link from the media service node to the receiving end, thereby reducing the round-trip time (RTT) required for detection and quickly converging to obtain the target MTU value.

[0075] At this point, if the receiving end determines that the largest MTU value among all the MTU values ​​to be tested is an available MTU value, then the largest MTU value to be tested can be directly used as the target MTU value for sending multimodal service data.

[0076] If the maximum MTU value to be tested is determined to be an unavailable MTU value, then the largest MTU value among the available MTU values ​​contained in each MTU value to be tested is taken as the first boundary MTU value, and the smallest MTU value among the unavailable MTU values ​​contained in each MTU value to be tested is taken as the second boundary MTU value. Then, a secondary probing process can be performed based on the first boundary MTU value and the second boundary MTU value, and the target MTU value can be determined based on the secondary probing results.

[0077] The aforementioned secondary detection process is used to select an intermediate value between the first boundary MTU value and the second boundary MTU value as the target MTU value.

[0078] Specifically, the sending end can determine at least one alternative MTU value from a preset pool of alternative MTU values ​​that lies between the first boundary MTU value and the second boundary MTU value, as the secondary test MTU value. Then, for each secondary test MTU value, it can construct secondary probe packets and send these packets in parallel to the receiving end. Finally, it selects the maximum value from the secondary test MTU values ​​from which the receiving end returns corresponding acknowledgment information, as the target MTU value. In other words, the sending end can determine the available MTU values ​​contained in each secondary test MTU value based on the acknowledgment information returned by the receiving end, and then select the maximum value from these available MTU values ​​as the target MTU value.

[0079] Furthermore, if it is determined that all MTU values ​​to be tested are unusable MTU values, the sending end can determine whether each MTU value to be tested contains a preset minimum MTU value. If so, a link connectivity test is performed, and the target MTU value is determined based on the test results.

[0080] Specifically, if the sending end determines from the test results that the global link is physically connected but there is ICMP packet filtering behavior, then the preset security MTU value is used as the target MTU value. If the test results show that the global link is not physically connected, then a link fault alarm is generated, and the minimum MTU value to be tested is used as the target MTU value.

[0081] In link connectivity testing, the sending end can determine the target MTU value by sending a specified probe packet with a Time To Live (TTL) value of 1 and based on the response result of the specified probe packet.

[0082] Specifically, when a probe packet with a TTL value of 1 leaves the sender and passes through the first intermediate network device (such as a router, gateway, or firewall), the device decrements the packet's TTL value by 1, making it 0. According to the IP protocol specification, when the packet's TTL value becomes 0, the intermediate network device must discard the packet and return an ICMP message of type 11 with code 0 to the sender, which is an ICMP message used to indicate "timeout". If the sending end determines, based on the response result, that it has received an ICMP message returned by an intermediate network device, it indicates that the probe packet has successfully reached at least one intermediate network device in the global link, and that the intermediate network device has normal ICMP response capability. Therefore, it can be determined that the link is physically connected but exhibits ICMP message filtering behavior (i.e., the intermediate device discards large packets without returning error information, but responds with a TTL expired message). In this case, it can be determined that the current network environment belongs to an ICMP black hole scenario. The sending end can select a secure transmission threshold (e.g., 1280) as the target MTU value to attempt to maintain communication. If the sending end determines, based on the response result, that it has not received an ICMP message returned by an intermediate network device or an acknowledgment message returned by the receiving end within a preset specified time period, it indicates that the probe packet failed to trigger feedback from any intermediate node. Therefore, it can be determined that the path is interrupted. In this case, the sending end can select the minimum MTU value to be tested as the target MTU value and generate a link failure alarm to prompt the user to check the network connection.

[0083] In practical applications, sending too many probe packets corresponding to the MTU value to be tested in parallel may lead to network congestion, bandwidth waste, and excessive terminal processing load. On the other hand, sending too few probe packets corresponding to the MTU value to be tested in parallel may result in insufficient detection accuracy. Therefore, in order to ensure detection accuracy while minimizing the number of probe packets to reduce bandwidth consumption and detection time, the number of each MTU value to be tested in this specification can be three, namely, the first preset value, the second preset value, and the third preset value.

[0084] The first preset value is the maximum MTU value determined based on historical network link MTU distribution statistics and protocol standards, such as 1500; the second preset value is the intermediate MTU value determined based on historical link overhead in each tunnel encapsulation or encrypted transmission scenario, such as 1400; and the third preset value is the minimum MTU value determined based on historical network link MTU distribution statistics and protocol standards, such as 576.

[0085] To facilitate understanding, the following example uses only the three specific MTU values ​​mentioned above as examples to explain in detail the process of determining the target MTU value in this specification. Figure 4 As shown.

[0086] Figure 4This is a schematic diagram of the process for determining the target MTU value provided in an exemplary embodiment.

[0087] Combination Figure 4 It can be seen that after the sending end sends probe packets corresponding to the first, second, and third preset values ​​in parallel, it can enter the response analysis stage. Based on the received confirmation information or local link status anomaly information, it can then execute the following branch logic to determine the target MTU value: In the first case, if the first preset value (1500 bytes) is determined to be an available MTU value, it indicates that the current global link supports the standard Ethernet MTU, and the sending end directly selects 1500 as the target MTU value.

[0088] In the second scenario, if the first preset value (1500 bytes) is determined to be an unusable MTU value, and the second preset value is a usable MTU value, the sending end can execute a secondary probing process.

[0089] Specifically, the sending end can select an intermediate value between 1400 and 1500 from a preset set of alternative MTU values ​​(such as 1492, 1476, 1450, etc. in the figure, which correspond to the typical MTU of common tunnel encapsulations such as PPPoE, GRE, and VXLAN) as the secondary test MTU value, and generate and send a secondary probe packet based on these secondary test MTU values.

[0090] Subsequently, if the sending end determines that there is a usable MTU value among the secondary test MTU values, it can select the largest usable MTU value as the target MTU value and return it; if all secondary test MTU values ​​are unusable MTU values, the second preset value is used as the target MTU value to ensure transmission stability.

[0091] In the third scenario, if both the first and second preset values ​​are unavailable MTU values, but the third preset value is an available MTU value, it indicates that the link has a severe MTU limitation or is in a low-bandwidth / high-overhead network environment. In this case, in order to balance connectivity and a certain transmission efficiency, the sender can select a fourth preset value between the third and second preset values ​​as the target MTU value (i.e., the fourth preset value is greater than the third preset value and less than the second preset value).

[0092] The fourth preset value mentioned above is a secure transmission threshold determined based on historical network link MTU distribution statistics and protocol standards (such as 1280 in the figure, which is the minimum MTU requirement of IPv6).

[0093] In practical applications, to ensure the accuracy of the target MTU value setting and prevent direct connection failures due to network fluctuations or special routing strategies, the sending end can generate a probe packet according to the fourth preset value before using it as the target MTU value. This probe packet is then sent to the receiving end. Ultimately, based on the local link status anomaly information returned by the intermediate network devices in the first and second links after the probe packet is sent, it can be determined whether to use the fourth preset value as the target MTU value. That is, if an acknowledgment is received from the receiving end, it is determined that the fourth preset value is actually usable on the global link, and it can be used as the target MTU value. Otherwise, the fourth preset value is abandoned, and the third preset value is used as the target MTU value to ensure basic connectivity.

[0094] In the fourth scenario, if the third preset value is determined to be an unusable MTU value, it indicates that there may be an anomaly in the global link. In this case, the sending end can perform a link connectivity test and determine the target MTU value based on the test results.

[0095] As can be seen from the above, the sending end can quickly determine the target MTU value in two round trips within a variety of complex scenarios, including standard networks, tunnel-encapsulated networks, restricted mobile networks, and anomalous black hole networks, by using parallel probing based on the three key MTU values ​​to be tested and subsequent conditional branching processing.

[0096] It is important to emphasize that in real-world multimodal real-time interactive applications, the network environment in which users operate is highly dynamic and unstable. For example, a user may move from an indoor Wi-Fi environment to an outdoor environment, causing the network connection to switch from a wireless LAN to a mobile communication network, or causing the network connection to switch between different base stations, or even, under the same network standard, changes in signal strength may alter the underlying routing path. This is because the MTU (Mean Transmission Unit) limitations of different network types (such as Wi-Fi and cellular networks) and different operator networks often differ significantly (for example, some Wi-Fi environments support a standard MTU of 1500 bytes, while some 4G / 5G networks or networks with specific tunnel encapsulation may only support 1400 bytes or less).

[0097] Therefore, in this specification, the transmitting end is also equipped with a full-link status monitoring module, which is used to monitor the current network connection status of the transmitting end in real time or periodically. When the link status monitoring module detects a network link switching event at the transmitting end (e.g., switching from Wi-Fi to 5G, or from 4G to Wi-Fi), it can determine that the currently determined target MTU value may be invalid. At this time, the transmitting end can trigger an adaptive reprobing procedure. In this procedure, the transmitting end can re-execute the aforementioned target MTU value detection steps to re-determine the target MTU value.

[0098] As described above, the link status monitoring module can monitor the current network connection status of the sending end by means of methods such as listening to network interface change broadcasts issued by the operating system (e.g., the ConnectivityManager callback in Android system, the NWPathMonitor status update in iOS system), detecting changes in IP addresses, monitoring changes in the default gateway, or periodically sending lightweight heartbeat probe packets to evaluate the connectivity and characteristic parameters of the current link.

[0099] It is worth noting that, in order to ensure service continuity and avoid data transmission interruption during reprobing at the moment of link switching, the sending end can monitor the network link switching event of the sending end. When a network link switching event is detected, the transmission MTU value of the service data is rolled back to the preset safe MTU value, and the target MTU value is re-determined.

[0100] Specifically, upon triggering a reprobing, the sender can temporarily fall back to a preset MTU value (such as 576 bytes or 1280 bytes, which can be set according to actual needs) to send multimodal service data. Since this preset MTU value is the minimum unfragmented size supported by various network environments, it can ensure that service data packets are not dropped or fragmented even before the specific target MTU value of the new link is determined, thereby ensuring that audio and video calls or AI interactions are not interrupted.

[0101] In one embodiment, the multimodal interaction system can have multiple receivers, and the media service node can establish independent transmission connections with each receiver, thereby forming a first link and multiple second links. In this case, the sender can simultaneously distribute multimodal service data to multiple receivers through the media service node.

[0102] Furthermore, since the second links corresponding to different receivers may be located in different network environments—for example, some receivers may be located in Wi-Fi networks, some in 4G / 5G mobile networks, or some second links may pass through different tunnel encapsulation, NAT, or firewall devices—the maximum unfragmented MTU value supported by each second link may differ.

[0103] In this multi-receiver scenario, the sender can still determine at least one MTU value to be tested in the aforementioned manner and construct a probe packet configured to not allow fragmentation. For each MTU value to be tested, the sender can initiate probes separately for multiple receivers via a media service node to perform full-link probing of the first link and the corresponding second links for each receiver. The media service node can send the probe packet to each receiver separately, or construct a corresponding downlink probe packet based on the MTU value to be tested and send it to each receiver to trigger the MTU probing process on each second link.

[0104] During the probe, if an intermediate network device included in a second link returns a local link status anomaly, this anomaly information can be received by the media serving node. The media serving node can parse the ICMP message or other feedback information indicating link MTU anomalies returned by the second link, generate application-layer feedback signaling associated with the corresponding receiver identifier, and then send the application-layer feedback signaling to the sender. Correspondingly, if a receiver successfully receives the corresponding probe packet, the receiver can return an acknowledgment, which the media serving node then forwards to the sender, or the sender receives the acknowledgment via the media serving node.

[0105] Based on this, on the transmitting end, the second link probe results corresponding to each receiving end can be maintained separately, and the probe results of the first link can be correlated with the probe results of each second link. For any MTU value to be tested, if the first link or any second link reports a partial link status anomaly, it can be determined that the MTU value to be tested is unusable for the corresponding full-link transmission path; if, for a certain receiving end, the first link does not report a partial link status anomaly, and the receiving end returns an acknowledgment, it can be determined that the MTU value to be tested is a usable MTU value for the full-link transmission path corresponding to that receiving end.

[0106] Furthermore, when the sending end needs to send the same service data to multiple receiving ends using a uniform packet size, the sending end can comprehensively judge the end-to-end probing results for multiple receiving ends and select the largest commonly supported MTU value from the available MTU values ​​for each receiving end as the target MTU value. In other words, the sending end can determine the candidate target MTU values ​​for each receiving end separately and select the minimum value from the candidate target MTU values ​​as the target MTU value when uniformly sending service data to multiple receiving ends. This ensures that the service data packets sent by the sending end can simultaneously adapt to the first link and the second link corresponding to each receiving end, thereby avoiding fragmentation or packet loss in the path corresponding to any receiving end due to an excessively large MTU value.

[0107] Furthermore, if the media service node or sender determines that the link capabilities of multiple receivers differ significantly, the multiple receivers can be divided into at least two receiver groups based on the candidate target MTU values ​​corresponding to each receiver, and a target MTU value can be determined for each receiver group. In this case, the sender can use different packet sizes to send service data for different receiver groups, or the media service node can perform differentiated forwarding based on the target MTU values ​​of different receiver groups.

[0108] Therefore, in multi-receiver scenarios, the transmitter can not only determine the target MTU value by combining the detection results of the first link and a single second link, but also perform multi-channel collaborative judgment by combining the detection feedback information corresponding to multiple second links to determine the target MTU value applicable to multiple receivers for common transmission, or to determine the target MTU value applicable to different receiver groups. This enables the solution provided in this specification to adapt to real-time interactive scenarios with multiple receivers, such as multi-person video conferencing, multi-person interactive live streaming, and multi-user cloud gaming.

[0109] As can be seen from the above, the sending end can send at least one probe packet configured to disallow fragmentation and perform global link probing on the first link from the sending end to the media service node and the second link from the media service node to the receiving end. When receiving local link status anomaly information from either the first or second link, it can be determined that the current MTU value to be tested exceeds the carrying capacity of that link, thus marking it as an unusable MTU value. Conversely, if receiving confirmation information from the receiving end, it is marked as an usable MTU value. Finally, based on the usable MTU value, the maximum unfragmented MTU value supported by all links in the entire transmission path from the sending end through the media service node to the receiving end can be determined as the target MTU value. This avoids problems such as data packets being forcibly fragmented at intermediate nodes due to an excessively large MTU value, leading to fragment loss, or insufficient bandwidth utilization due to an excessively small MTU value.

[0110] To facilitate understanding, the methods executed by the media service node during multimodal data transmission are explained in detail below, as follows: Figure 5 As shown.

[0111] Figure 5 This is a schematic flowchart of a multimodal data transmission method provided in an exemplary embodiment, including the following steps: S500: Receive a probe packet sent by the sending end and intended for the receiving end, wherein the probe packet is created according to the maximum transmission unit (MTU) value to be tested and is configured to not allow fragmentation. S502: Send the probe packet to the receiving end to probe the second link; S504: If an Internet Control Message Protocol (ICMP) message is received from an intermediate network device included in the second link, the ICMP message is parsed to generate local link state anomaly information for the second link and returned to the sending end, so that the sending end can determine the MTU value to be tested as an unusable MTU value based on the local link state anomaly information. S506: If the receiving end receives the confirmation information, then forward the confirmation information to the sending end so that the sending end can determine the MTU value to be tested as an available MTU value based on the confirmation information.

[0112] As can be seen from the above, the media service node can act as a relay and coordination node between the sender and receiver during multimodal data transmission. On the one hand, it receives probe packets sent by the sender with the receiver as the receiving end, and sends the probe packets to the receiver to probe the second link from the media service node to the receiver. On the other hand, when an abnormal feedback occurs in the second link due to the MTU value under test exceeding the link carrying capacity, the media service node can receive the corresponding ICMP message, parse the ICMP message, generate local link state abnormality information to characterize the abnormality of the second link, and return it to the sender, so that the sender can combine the probe results of the first and second links to determine whether the current MTU value under test is usable.

[0113] In addition, when the receiving end successfully receives the probe packet and returns an acknowledgment message, the media service node can also forward the acknowledgment message to the sending end, so that the sending end can determine the corresponding MTU value to be tested as an available MTU value.

[0114] Therefore, the media service node can provide the sender with second-link detection feedback without changing the sender's role as the subject of target MTU value determination. This enables the sender to determine the target MTU value of the entire link from the sender through the media service node to the receiver, for use in the subsequent transmission of multimodal service data.

[0115] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 6As shown, device 600 mainly consists of a communication interface 602, a user interface 604, a processor 606, and a data storage 608. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 610. The communication interface 602 enables device 600 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 602 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 602 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 602 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 602 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0116] User interface 604 includes receiving user input and providing output to the user. Therefore, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 604 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 604 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 600 may support remote access from other devices via communication interface 602 or another physical interface (not shown). User interface 604 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 604 may also be configured as a display device for rendering or displaying text fragments.

[0117] Processor 606 may contain one or more general-purpose processors and / or special-purpose processors.

[0118] Data storage 608 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 606. Data storage 608 may include removable and non-removable components.

[0119] Processor 606 is capable of executing program instructions 618 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 608 to perform the various functions described herein. Data storage 608 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 600, enable device 600 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 618 by processor 606 may result in processor 606 using data 612.

[0120] For example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 600 and one or more applications 620 (e.g., a browser, social application, or game application). Similarly, data 612 may include operating system data 616 and application data 614. Operating system data 616 is primarily accessible to the operating system 622, while application data 614 is primarily accessible to one or more applications 620. Application data 614 may reside in a file system visible or hidden from the user of device 600.

[0121] Application 620 can communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs help application 620 read and / or write application data 614, transmit or receive information via communication interface 602, receive or display information on user interface 604, etc.

[0122] In some terminology, application 620 may be simply referred to as "app". Furthermore, application 620 can be downloaded to device 600 through one or more online app stores or app markets. However, applications can also be installed on device 600 in other ways, such as through a web browser or a physical interface on device 600 (e.g., a USB port).

[0123] Please refer to Figure 7 Multimodal data transmission devices can be applied to, for example... Figure 6 The device shown implements the technical solution of this specification. The multimodal data transmission apparatus may include: The determination module 701 is used to determine the maximum transmission unit (MTU) value to be tested and to create a probe packet configured to not allow fragmentation according to the MTU value to be tested. The detection module 702 is used to send the detection packet to the receiving end as the receiving object, so as to detect the first link and the second link; The adjustment module 703 is configured to determine that the MTU value to be tested is an unusable MTU value if it receives local link status abnormality information of the first link returned by an intermediate network device included in the first link, or receives local link status abnormality information of the second link returned by the media service node; and to determine that the MTU value to be tested is an usable MTU value when it receives confirmation information returned by the receiving end. The service data transmission module 704 is used to select a target MTU value based on the available MTU value; and to send service data based on the target MTU value.

[0124] Optionally, the sending end has a service socket and a listening socket pre-created; The detection module 702 is specifically configured to: send the detection packet to the receiving end via the service socket to detect the first link and the second link; and receive local link status anomaly information returned by the media service node via the service socket; the local link status anomaly information is application layer feedback signaling generated by the media service node after parsing the Internet Control Message Protocol (ICMP) messages returned by the intermediate network devices included in the second link; and capture the local link status anomaly information returned by the intermediate network devices included in the first link via the listening socket.

[0125] Optionally, the detection module 702 is specifically used to determine the network data access permissions of the operating system corresponding to the sending end; if the operating system grants access permissions to the underlying network data, it intercepts network layer IP packets through the listening socket, parses and extracts local link state anomaly information; if the operating system restricts access permissions to the underlying network data, it calls the standard network state listening interface provided by the operating system through the listening socket to listen for transport layer state feedback events in order to obtain the local link state anomaly information.

[0126] Optionally, the detection module 702 is specifically used to, when there are multiple MTU values ​​to be tested, construct a detection packet with a length equal to the MTU value to be tested for each MTU value to be tested; and send each detection packet in parallel with the receiving end as the receiving object to detect the first link and the second link.

[0127] Optionally, the service data transmission module 704 is specifically configured to: when determining that the largest MTU value among the MTU values ​​to be tested is an available MTU value, use the largest MTU value to be tested as the target MTU value for sending multimodal service data; or, when determining that the largest MTU value to be tested is an unavailable MTU value, use the largest MTU value among the available MTU values ​​as the first boundary MTU value and the smallest MTU value among the unavailable MTU values ​​as the second boundary MTU value; perform a secondary probing process based on the first boundary MTU value and the second boundary MTU value, and determine the target MTU value based on the secondary probing results; the secondary probing process is used to select an intermediate value between the first boundary MTU value and the second boundary MTU value as the target MTU value.

[0128] Optionally, the service data transmission module 704 is specifically configured to: determine at least one alternative MTU value located between the first boundary MTU value and the second boundary MTU value from a preset set of alternative MTU values, as a secondary test MTU value; construct each secondary probe packet for each secondary test MTU value, and send each secondary probe packet to the receiving end in parallel; select the maximum value from each secondary test MTU value that receives the corresponding confirmation information returned by the receiving end, as the target MTU value.

[0129] Optionally, the at least one MTU value to be tested includes: a first preset value, which is the maximum MTU value determined based on historical network link MTU distribution statistics and protocol standards; a second preset value, which is the intermediate MTU value determined based on historical link overhead in each tunnel encapsulation or encrypted transmission scenario; and a third preset value, which is the minimum MTU value determined based on historical network link MTU distribution statistics and protocol standards.

[0130] Optionally, the service data transmission module 704 is specifically configured to, when it is determined that the first preset value and the second preset value are both unusable MTU values, and the third preset value is an available MTU value, use the fourth preset value as the target MTU value; the fourth preset value is greater than the third preset value and less than the second preset value; the fourth preset value is a secure transmission threshold determined based on historical network link MTU distribution statistics and protocol standards.

[0131] Optionally, the service data transmission module 704 is specifically configured to: if it is determined that each MTU value to be tested is an unusable MTU value, then determine whether each MTU value to be tested contains a preset minimum MTU value; if so, then send a specified probe packet to the receiving end as the receiving object, and determine the target MTU value according to the response result of the intermediate network device to the specified probe packet; the time-to-live value of the specified probe packet is set to 1.

[0132] Optionally, the service data transmission module 704 is specifically used to: when it is determined from the response result that the link is physically connected but there is ICMP message filtering behavior, use a preset security MTU value as the target MTU value; when it is determined from the response result that the path is interrupted, generate a link fault alarm and use the minimum MTU value to be tested as the target MTU value.

[0133] Optionally, the detection module 702 is specifically used to send service data packets corresponding to the service data in parallel while sending the detection packet to the receiving end according to the MTU value to be tested; the service data packets are divided according to a preset initial MTU value.

[0134] Optionally, the determining module 701 is further configured to monitor network link switching events at the sending end; when the network link switching event is detected, the transmission MTU value of the service data is rolled back to a preset safe MTU value, and the target MTU value is redefined.

[0135] Please refer to Figure 8 Multimodal data transmission devices can be applied to, for example... Figure 6 The device shown implements the technical solution of this specification. The multimodal data transmission apparatus may include: The receiving module 801 is used to receive probe packets sent by the sending end and intended to be received by the receiving end. The probe packets are created according to the maximum transmission unit (MTU) value to be tested and are configured to not be fragmented. The detection module 802 is used to send the detection packet to the receiving end to detect the second link; The generation module 803 is used to parse the Internet Control Message Protocol (ICMP) message returned by the intermediate network device included in the second link if it receives the ICMP message, generate local link state anomaly information of the second link and return it to the sending end, so that the sending end can determine the MTU value to be tested as an unusable MTU value based on the local link state anomaly information. The forwarding module 804 is used to forward the confirmation information to the sending end when it receives the confirmation information returned by the receiving end, so that the sending end can determine the MTU value to be tested as an available MTU value based on the confirmation information.

[0136] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0137] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0138] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0139] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0140] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0141] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0142] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0143] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0144] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0145] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0146] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A multimodal data transmission method, the method being applied to the transmitting end of a multimodal interaction system, the multimodal interaction system comprising: The method comprises: a sending end, a media service node, and a receiving end; the sending end communicating with the media service node via a first link; and the receiving end communicating with the media service node via a second link. Determine the maximum transmission unit (MTU) value to be tested, and create a probe packet configured to not allow fragmentation according to the MTU value to be tested; The probe packet is sent to the receiving end to probe the first link and the second link; If the intermediate network device included in the first link returns a local link status abnormality information of the first link, or if the media service node returns a local link status abnormality information of the second link, then the MTU value to be tested is determined to be an unusable MTU value. If an acknowledgment is received from the receiving end, the MTU value to be tested is determined to be an available MTU value. Based on the available MTU value, a target MTU value is selected; and service data is sent based on the target MTU value.

2. The method as described in claim 1, wherein the sending end has a pre-created service socket and a listening socket; Sending the probe packet to the receiving end to probe the first link and the second link, specifically including: The probe packet is sent to the receiving end via the service socket to probe the first link and the second link; and the local link status anomaly information returned by the media service node is received via the service socket; the local link status anomaly information is application layer feedback signaling generated by the media service node after parsing the Internet Control Message Protocol (ICMP) message returned by the intermediate network device included in the second link. The listening socket captures local link status anomaly information returned by intermediate network devices included in the first link.

3. The method as described in claim 2, wherein capturing local link status anomaly information returned by intermediate network devices included in the first link through the listening socket specifically includes: Determine the network data access permissions of the operating system corresponding to the sending end; If the operating system grants access to underlying network data, then network layer IP packets can be intercepted through the listening socket, and local link status anomaly information can be parsed and extracted. If the operating system restricts access permissions to underlying network data, the standard network status monitoring interface provided by the operating system is invoked through the listening socket to listen for transport layer status feedback events in order to obtain information about the abnormal status of the local link.

4. The method as described in claim 1 or 2, wherein the probe packet is sent to the receiving end as the receiving target to probe the first link and the second link, specifically including: When there are multiple MTU values ​​to be tested, a probe packet with a length equal to that MTU value is constructed for each MTU value to be tested. The receiving end is used as the receiving target to send each probe packet in parallel to probe the first link and the second link.

5. The method as described in claim 4, wherein a target MTU value is selected based on the available MTU value, specifically including any one of the following: If the largest MTU value among all the MTU values ​​to be tested is determined to be a usable MTU value, then the largest MTU value to be tested is used as the target MTU value for transmitting multimodal service data; or If the maximum MTU value to be tested is determined to be an unavailable MTU value, then the maximum value among all available MTU values ​​is taken as the first boundary MTU value, and the minimum MTU value among all unavailable MTU values ​​is taken as the second boundary MTU value. Based on the first boundary MTU value and the second boundary MTU value, a secondary detection process is performed, and based on the secondary detection results, the target MTU value is determined.

6. The method as described in claim 5, wherein a secondary detection process is performed based on the first boundary MTU value and the second boundary MTU value, and the target MTU value is determined based on the secondary detection result, specifically including: From the preset candidate MTU values, at least one candidate MTU value located between the first boundary MTU value and the second boundary MTU value is determined as the secondary test MTU value; For each secondary test MTU value, construct each secondary probe packet and send each secondary probe packet to the receiving end in parallel; The maximum value is selected from the secondary test MTU values ​​that receive the corresponding confirmation information from the receiving end, and is taken as the target MTU value.

7. The method of claim 6, wherein the at least one MTU value to be tested comprises: The first preset value is the maximum MTU value determined based on historical network link MTU distribution statistics and protocol standards. The second preset value is the intermediate MTU value determined based on the historical link overhead under each tunnel encapsulation or encrypted transmission scenario; The third preset value is the minimum MTU value determined based on historical network link MTU distribution statistics and protocol standards.

8. The method of claim 7, wherein selecting a target MTU value based on the available MTU value specifically includes: If it is determined that the first preset value and the second preset value are both unusable MTU values, and the third preset value is an usable MTU value, then the fourth preset value is taken as the target MTU value. The fourth preset value is greater than the third preset value and less than the second preset value; the fourth preset value is a secure transmission threshold determined based on historical network link MTU distribution statistics and protocol standards.

9. The method as described in claim 1, wherein selecting a target MTU value based on the available MTU value specifically includes: If it is determined that each MTU value to be tested is an unusable MTU value, then determine whether the MTU values ​​to be tested contain a preset minimum MTU value. If so, a specified probe packet is sent to the receiving end, and the target MTU value is determined based on the response of the intermediate network device to the specified probe packet; the time to live value of the specified probe packet is set to 1.

10. The method of claim 9, wherein determining the target MTU value based on the response result specifically includes: If the response result indicates that the link is physically connected but ICMP packet filtering behavior exists, then the preset security MTU value will be used as the target MTU value. If the path is determined to be interrupted based on the response result, a link failure alarm is generated, and the minimum MTU value to be tested is used as the target MTU value.

11. The method of claim 1, further comprising: During the process of sending probe packets to the receiving end according to the MTU value to be tested, service data packets corresponding to the service data are sent in parallel. The service data packets are divided according to a preset initial MTU value.

12. The method of claim 1, further comprising: Monitor the network link switching events at the sending end; When the network link switching event is detected, the transmission MTU value of the service data is rolled back to the preset safe MTU value, and the target MTU value is redefined.

13. A multimodal data transmission method, the method being applied to a media service node of a multimodal interaction system, the multimodal interaction system comprising: The method comprises a sending end, the media service node, and a receiving end, wherein the sending end communicates with the media service node via a first link, and the receiving end communicates with the media service node via a second link. Receive probe packets sent by the sending end and intended for the receiving end, wherein the probe packets are created according to the maximum transmission unit (MTU) value to be tested and are configured to not fragment. The probe packet is sent to the receiving end to probe the second link; If an Internet Control Message Protocol (ICMP) message is received from an intermediate network device included in the second link, the ICMP message is parsed to generate local link state anomaly information for the second link and returned to the sending end, so that the sending end can determine the MTU value to be tested as an unusable MTU value based on the local link state anomaly information. If the receiving end receives the confirmation information, it forwards the confirmation information to the sending end so that the sending end can determine the MTU value to be tested as an available MTU value based on the confirmation information.

14. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-13 by executing the executable instructions.

15. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-13.

16. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-13.

Images