Media transmission method and computer device

Abstract

This specification provides a media transmission method and a computer device, wherein the media transmission method applied to a client includes: obtaining configuration information from a server; the configuration information being information included in a message returned by the server that the client needs to receive during connection negotiation with the server; performing connection negotiation, intercepting a first message sent to the server, and constructing a second message returned by the server in response to the first message based on the configuration information to complete the connection negotiation; constructing and caching a first instance based on the connection negotiation result; and performing media transmission based on the cached first instance and the server when media transmission with the server is required.

Description

Technical Field

[0001] This specification relates to the field of data transmission technology, and more particularly to a media transmission method and computer device. Background Technology

[0002] With the development of technologies such as artificial intelligence and cloud computing, an increasing number of scenarios require media transmission between user clients and service servers, such as transmitting multimedia data like audio, video, or video feeds. For example, when a user uses an AI dialogue function, the client needs to send the recorded audio of the user's speech to the server so that the server can determine the content of the user's speech and respond based on the AI. Similarly, when a user uses a cloud gaming service, the server needs to send the game feed generated by the game running on the server to the client so that the user can play cloud games through the client.

[0003] When there is a need to transmit multimedia data between a client and a server, the first step is to establish a Web Real-Time Communications (WebRTC) connection. A WebRTC connection is an end-to-end secure real-time transmission channel established through connection negotiation, used for low-latency sending and receiving of multimedia data between the two ends. Connection negotiation is used to negotiate what is transmitted between the two ends, how the connection is established, and how encryption is performed.

[0004] In existing technologies, there is a significant delay in the WebRTC connection establishment process between the client and the server. Summary of the Invention

[0005] In view of the above, this specification provides a media transmission method and a computer device through one or more embodiments.

[0006] According to a first aspect of one or more embodiments of this specification, a media transmission method is proposed, applied to a client, comprising:

[0007] Obtain configuration information from the server; the configuration information is the information included in the message returned by the server that the client needs to receive during connection negotiation with the server.

[0008] The connection negotiation is performed by intercepting the first message sent to the server and constructing the second message returned by the server in response to the first message based on the configuration information, so as to complete the connection negotiation.

[0009] Based on the connection negotiation results, construct and cache the first instance;

[0010] In cases where media transmission with the server is required, media transmission is performed based on the first cached instance and the server.

[0011] According to a second aspect of one or more embodiments of this specification, a media transmission apparatus is provided for use on a client, comprising:

[0012] The configuration information acquisition module is used to obtain configuration information from the server; the configuration information is the information included in the message returned by the server when the client and the server negotiate a connection.

[0013] The connection negotiation module is used to perform connection negotiation, intercept the first message sent to the server, and construct the second message returned by the server in response to the first message according to the configuration information to complete the connection negotiation.

[0014] The instance caching module is used to build and cache the first instance based on the connection negotiation result;

[0015] The activation module is used to perform media transmission based on the cached first instance and the server when media transmission with the server is required.

[0016] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the media transmission method as described in the first aspect of the embodiments of this specification.

[0017] According to a fourth aspect of the embodiments of this specification, a computer device is provided, the computer device comprising:

[0018] processor;

[0019] Memory used to store processor-executable instructions;

[0020] The processor implements the media transmission method as described in the first aspect of the embodiments of this specification by running the executable instructions.

[0021] According to a fifth aspect of the embodiments of this specification, a computer program product is provided that, when executed by a processor, implements the media transmission method as described in the first aspect of the embodiments of this specification.

[0022] In one or more embodiments of this specification, during the preparation phase, configuration information is obtained from the server in advance. Then, during connection negotiation, the configuration information is used to simulate the messages sent by the server during connection negotiation, thereby pre-completing connection negotiation on the client side and generating and caching a first instance that can be used for media transmission. In this way, when multimedia data needs to be transmitted with the server, the cached first instance can be used to quickly respond to user requests and send multimedia data. This allows the client to respond faster when the user needs to use services provided by the server, improving the user experience.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0025] Figure 1 This is a flowchart of a WebRTC connection establishment method.

[0026] Figure 2 This is an application scenario diagram of a media transmission method.

[0027] Figure 3 This is a flowchart of a media transmission method.

[0028] Figure 4 It is a hardware structure diagram of a computer device. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0030] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0031] To facilitate understanding, the WebRTC connection establishment process will be explained first, and its flowchart is as follows: Figure 1 As shown.

[0032] First, after a user initiates a service requiring media transmission with the server, the client first creates a PeerConnectionFactory locally. This PeerConnectionFactory initializes and provides the ability to create various WebRTC objects, especially PeerConnections (PCs). A PeerConnection is a concrete connection instance responsible for media negotiation, Interactive Connectivity Establishment (ICE) connection, Datagram Transport Layer Security (DTLS) handshake, and Secure Real-time Transport Protocol (SRTP) encryption.

[0033] Next, the client creates a peering connection instance based on the peering connection factory. During instance creation, connection policies and network parameters are set, such as which STUN / TURN servers to use and transmission / multiplexing strategies. Event callbacks are also registered within the peering connection instance to receive notifications such as candidate address generation, remote media arrival, and connection status changes. The peering connection instance also configures media transmission and reception, starts media acquisition devices, and binds the multimedia data tracks to be sent.

[0034] In peer-to-peer connection instances, SDP negotiation and ICE negotiation are performed. SDP negotiation is mainly used for the two communicating parties to exchange candidate addresses, while ICE negotiation is mainly used to determine a passable network path based on the exchanged candidate addresses.

[0035] In practical implementation, assuming the two parties needing to establish a WebRTC connection are A and B, then A might first create a Session Description Protocol (SDP) offer. Creating an SDP offer typically triggers the ICE candidate address collection process, gathering candidate addresses that A can use for media transmission. After collecting the candidate addresses, A adds this information to the SDP offer message and sends it to B. Furthermore, the SDP offer message and SDP response message (hereinafter collectively referred to as SDP messages) also include media-related information, such as the supported codec formats (to facilitate negotiation between the two parties for a mutually applicable codec format). The SDP message can also include ICE options and DTLS-related information to facilitate ICE negotiation and the DTLS handshake, and to prevent man-in-the-middle attacks during the DTLS handshake phase.

[0036] Similarly, B-end collects its own available candidate addresses based on the ICE candidate collection, adds the collected candidate addresses and other information to its SDP answer message, and returns it to A-end. In addition to the candidate addresses, B-end's SDP answer message also includes information about the transmission media mentioned above, as well as ICE options and DTLS related information. Based on the received SDP answer, A-end can determine that SDP negotiation is complete. Thus, through SDP negotiation, both ends obtain each other's candidate addresses that can be used for media transmission.

[0037] After both ends obtain each other's candidate addresses through SDP negotiation and collect their own candidate addresses through ICE candidate address collection, ICE negotiation can proceed. This involves performing connectivity checks on the candidate addresses and selecting a pair of connectable paths. Once a connectable path is determined, the ICE negotiation is considered complete.

[0038] Furthermore, the above explanation uses the example of the candidate address being included in the SDP offer and response messages. In another alternative implementation, the candidate address may not be included in the SDP offer and response messages, but rather... Figure 1 As shown, after exchanging SDP offer and SDP response messages, candidate addresses are sent separately via signaling.

[0039] After completing ICE and SDP negotiations via a peer-to-peer connection, the two ends can perform a DTLS handshake to establish a secure session. During the DTLS handshake, keys for encryption and verification, and certificates for authentication, can be exchanged to establish a secure session.

[0040] The two ends of a WebRTC connection, as mentioned above, can be two users, such as two users making a video call through a WebRTC connection. Alternatively, the two ends can be a client and a server. For example, a user (corresponding to the client) might use an AI dialogue function provided by the server, or a user might use a cloud rendering function provided by the server (for scenarios requiring cloud rendering, such as cloud gaming), or a user might use a video customer service function provided by the server. In these scenarios, response speed is generally important. For instance, in scenarios where a connection is established between a client and a server, a fast response from the client is usually required to improve the user experience.

[0041] The methods mentioned above suffer from a lengthy connection establishment process. Specifically, creating a peer connection factory requires loading libraries and initializing threads, typically taking 50-150ms. Establishing a peer connection itself requires constructing a context and enumerating codecs, generally taking 20-50ms. The SDP negotiation process involves exchanging SDP offer and response messages, consuming 30-80ms. The ICE negotiation process requires collecting candidate addresses and performing connectivity checks, a process heavily influenced by network conditions, taking 100-500ms. Furthermore, obtaining video sources necessitates requesting user permissions and activating the device, typically taking 50-200ms.

[0042] As can be seen, the connection establishment process mentioned above takes a long time overall. The process of establishing a WebRTC connection usually takes more than 800ms, which cannot meet the requirements for fast response.

[0043] Based on this, this specification provides a media transmission method applied to a client. The method includes two phases: a preparation phase and an activation phase. In the preparation phase, configuration information required for connection negotiation is obtained from the server beforehand. Then, the server performs connection negotiation. During the negotiation process, the server intercepts a first message sent to it for connection negotiation and, based on the obtained configuration information, constructs a second message returned by the server in response to the first message to complete the connection negotiation. The server also caches a first instance for media transmission based on the connection negotiation result. In the activation phase, when media communication with the server is required, media transmission is performed based on the cached first instance.

[0044] During the preparation phase, configuration information is obtained from the server in advance. Then, during connection negotiation, this configuration information is used to mimic the messages sent by the server during connection negotiation. This allows the connection negotiation to be pre-completed on the client side, generating and caching a first instance that can be used for media transmission. Thus, when multimedia data needs to be transmitted with the server, the cached first instance can be used to quickly respond to user requests and send multimedia data. This results in a faster response time for the client when it needs to use services provided by the server, improving the user experience.

[0045] Furthermore, this specification addresses the client-server communication scenario. Considering that compared to the scenario where two clients establish a connection for media transmission, in the client-server connection establishment scenario, it is known in advance which party (i.e., the server) will be communicating with, and the configuration information (such as address, key, and encoding / decoding format) that the server needs to send during the connection negotiation process is fixed, the configuration information is obtained from the server in advance for the server-client connection establishment scenario. Therefore, during connection negotiation, this configuration information can be used to simulate the messages returned by the server, thereby pre-completing the client's connection negotiation and caching the first instance generated based on this connection negotiation.

[0046] The media transmission method shown in this specification will now be explained.

[0047] This method involves two parties: a client and a server. The client is the user's terminal, which can be a smartphone, smart wearable device, computer, or other personal terminal. The server is a server that provides services that require media transmission between the client and server. These services may include AI dialogue, cloud rendering, remote customer service, etc. Media transmission refers to the transmission of multimedia data between the two ends or from one end to the other. Multimedia data may include video data, audio data, etc., and the type and direction of the transmitted multimedia data depend on the application scenario of this method.

[0048] In an optional implementation, the server can be a Selective Forwarding Unit (SFU). An SFU server is used for relaying and forwarding multimedia streams from multiple users. In this scenario, the application scenarios described in this specification are as follows: Figure 2 As shown, the client and the SFU server communicate with each other, establishing a connection to transmit multimedia data. This connection can be a WebRTC connection.

[0049] The following section describes a media transmission method provided in this specification. The flowchart of this method is as follows: Figure 3 As shown. The method consists of two phases: a preparation phase and an activation phase. Figure 3 Steps 301, 303, and 305 correspond to the preparation phase, and step 307 corresponds to the activation phase. The following will discuss... Figure 3 Each step in the process will be explained.

[0050] First, let's explain the preparation phase. The preparation phase can be executed either after the user first launches the application or after the user navigates to the application from another interface on the terminal. By performing the preparation phase before any media transmission needs arise, the time spent on the connection establishment process can be reduced, improving the user experience.

[0051] Step 301: Obtain configuration information from the server.

[0052] The configuration information refers to the information included in the messages returned by the server that the client needs to receive during connection negotiation with the server.

[0053] Specifically, this step can involve obtaining configuration information from the server in advance via signaling services. Step 301 differs from the method used during connection negotiation. During connection negotiation, the client first sends a connection offer message (such as an SDP offer message), and the server responds with a connection response message. Furthermore, during connection negotiation, these messages must conform to the specifications of the corresponding protocol, and the configuration information may be obtained from messages corresponding to multiple protocols. Step 301, however, retrieves all configuration information from the server at once, and the transmission of this configuration information may not conform to the message format required for connection negotiation.

[0054] Configuration information may specifically include the server's address, the multimedia data encoding and decoding formats supported by the server, as well as keys and certificates. This information is what the server needs to send through the connection negotiation protocol during the connection establishment process.

[0055] Furthermore, in an optional implementation, the aforementioned connection negotiation can be completed through peer connections and peer connection factories. The meanings of peer connections and peer connection factories are detailed above and will not be repeated here. Therefore, before executing step 303, if peer connections and peer connection factories have not been created beforehand, a peer connection factory can be created first; based on the peer connection factory, a peer connection is created. Accordingly, step 303 can be connection negotiation based on the peer connection.

[0056] Furthermore, to facilitate the creation of instances for media transmission, media tracks need to be bound to these instances. In the case of creating a peer-to-peer factory, tracks need to be bound within the peer-to-peer connections. In an alternative implementation, a dummy media source can be created and bound to a media track to simulate a real media source in a first instance. This first instance is the one used for media transmission. This avoids initiating hardware capture on the client side, protecting user client privileges from abuse and safeguarding user privacy; it also reduces hardware resource consumption.

[0057] Step 303: Perform connection negotiation, intercept the first message sent to the server, and construct the second message returned by the server in response to the first message according to the configuration information to complete the connection negotiation.

[0058] Connection negotiation refers to the process where both ends agree on how to transmit multimedia data. The two ends negotiate using signaling services, following the protocol defined in the connection negotiation procedure. During the negotiation, the two parties exchange messages.

[0059] In a specific implementation of step 303, an optional implementation can be achieved using a hook mechanism. A hook mechanism is a scalable control technique that inserts custom interception points into the execution path of a target function / event to capture, redirect, or enhance its call flow (such as pre- / post-processing, parameter and return value modification). In step 303, a hook can be used to intercept the first message sent to the server and rewrite its implementation flow as follows: construct a second message returned by the server in response to the first message based on the configuration information. This allows interception of the negotiation message sent by the actual client, enabling the client to determine that the negotiation is complete.

[0060] In an optional implementation, the established connection may be a WebRTC connection, and the connection negotiation may include SDP negotiation and ICE negotiation. Correspondingly, the first message may include two types of messages: an SDP offer message and a first ICE message for ICE negotiation. Correspondingly, the second message may include an SDP response message and a second ICE message for ICE negotiation.

[0061] The specific implementation of step 303 can include: intercepting the SDP offer message sent to the server, and constructing an SDP response message returned by the server in response to the SDP offer according to the configuration information. As mentioned earlier, when the client negotiates a connection, it first sends an SDP offer message, which includes information related to the transmitted media, ICE options, and DTLS information. This information can be collected using the same methods as in related technologies, and will not be elaborated here. In the method of this specification, a connection is not actually established with the server during the preparation phase. Instead, the client intercepts the SDP offer message sent to the server and generates an SDP response message according to the configuration information. As mentioned earlier, the content of the SDP offer message and the SDP response message is similar, including media-related information (such as encoding / decoding format), and ICE options and DTLS information. The configuration information can include all or part of this information (if the configuration information only includes part of the information, the SDP response message can be constructed according to the configuration information and the default preset configuration), so the SDP response message can be constructed by mimicking the configuration information.

[0062] In one optional implementation, when generating the SDP offer message, the client's ICE candidate address collection process is triggered, that is, collecting its own candidate addresses for transmitting multimedia data. Accordingly, the SDP offer message includes the collected client candidate addresses, and an SDP response message including the server's candidate addresses is constructed based on the server's candidate addresses included in the configuration information.

[0063] In another alternative implementation, the SDP offer message does not include candidate addresses. After sending the SDP offer message and receiving the SDP response message, the client triggers the ICE candidate address collection process to gather its own candidate addresses for transmitting multimedia data. Then, via signaling, the client sends a message to the server containing its collected candidate addresses. The client intercepts this message using a hook mechanism or other methods and, based on the server's candidate addresses included in the configuration information, constructs a message returned by the server containing its own candidate addresses. This allows the client to confirm that candidate address exchange has been completed.

[0064] After exchanging candidate addresses, the two ends still need to perform ICE negotiation. The client can then generate a first ICE message triggered by the SDP offer message. As mentioned earlier, the first ICE message is used for connectivity detection, selecting connectable paths from multiple paths composed of candidate addresses from both ends. The candidate address pairs included in the first ICE message can be generated from the client's collected candidate addresses and the server's candidate addresses included in the configuration information.

[0065] The client can intercept the first ICE message sent to the server and construct a second ICE message returned by the server in response to the first ICE message. During this ICE negotiation process, the client and server do not actually negotiate; instead, the client intercepts the first ICE message and constructs the second ICE message. The first and second ICE messages are used for connectivity detection. When constructing the second ICE message, the client can assume that all candidate address pairs for connectivity detection are connected. The final candidate address pair selected for use can, in an optional implementation, be randomly selected from several candidate address pairs.

[0066] Furthermore, in an optional implementation where the SDP offer message does not include candidate addresses, the client can generate a first ICE message upon triggering the SDP offer message. This first ICE message differs from the first ICE message mentioned above; it refers to a message used for candidate address exchange via signaling, constructed based on local candidate address collection results. Correspondingly, the client can intercept this first ICE message and construct a second ICE message, which can be constructed based on the server's candidate addresses included in the configuration information. The second ICE message is used for candidate address exchange, returning available candidate addresses from the server to the client. Next, an ICE negotiation process, i.e., connectivity detection, can be performed; the specific implementation details are described above and will not be repeated here. In another optional implementation, a pair of addresses can be specified as candidate addresses for communication without performing the ICE connectivity detection process.

[0067] After completing the ICE negotiation, the ICE connection state (iceConnectionState) can be changed to "connected" based on the second ICE message.

[0068] After the SDP and ICE negotiations are completed, in an alternative implementation, as described above, both ends can perform a DTLS handshake to establish a secure session between the client and the server.

[0069] Specifically, the client can perform a DTLS handshake with the server. During the handshake, the client generates a handshake message (hereinafter referred to as the fourth handshake message). The handshake involves multiple rounds of message exchange, and the fourth handshake message can include several messages sent by the client to the server for the DTLS handshake. During the DTLS handshake, in order to establish a secure session, both ends need to exchange various types of data, such as certificates and keys. The certificate data included in the fourth handshake message can be collected locally.

[0070] As mentioned earlier, this step does not directly establish a connection with the server; instead, the client constructs the messages sent by the server. Specifically, the client intercepts the fourth handshake message sent to the server and constructs the fifth handshake message returned by the server based on the fourth handshake message to complete the DTLS handshake. Similar to the fourth handshake message, the fifth handshake message is used for the DTLS handshake and may include multiple messages for the DTLS handshake. The fifth handshake message can be generated based on configuration information. This configuration information may include the server's certificate and key (or a random number used to generate the key, etc.), thus allowing the fifth handshake message to be constructed based on the configuration information.

[0071] In a WebRTC connection, the connection state can be changed to "connected" based on the DTLS handshake result.

[0072] The above example, using WebRTC, illustrates the specific implementation of step 303. In step 303, the message sent from the client to the server is intercepted, and a message returned by the server is constructed based on pre-obtained configuration information. This allows the client to determine that the connection negotiation is complete based on the returned message, thus enabling the creation of an instance for media transmission. Furthermore, because the second message is constructed using pre-obtained configuration information, rather than directly interacting with the server to obtain the second message containing configuration information, the connection negotiation process is seamless for the server. The server does not need to communicate with the client during the connection negotiation process, nor does it need to prepare media transmission resources for the client during this process.

[0073] Furthermore, the reason why a connection is not actually established with the server in step 303 is primarily because: First, establishing a connection with the server would consume network resources during the connection establishment process. Moreover, after the connection is established, it is often necessary to measure the round-trip time (RTT) of the established path to determine its connectivity status, and RTT consumes significant network resources, resulting in resource waste. Second, if no multimedia data is sent for a period of time after the connection is established, the path may be disconnected, which would be detrimental to subsequent user access. Therefore, in this specification, the client constructs the message returned by the server to ensure that network resources are not wasted and that the instance is not released due to a prolonged period without multimedia data transmission.

[0074] Step 305: Based on the connection negotiation result, construct and cache the first instance.

[0075] Specifically, after completing the connection negotiation through the preceding steps, a pre-created first instance for media transmission can be obtained, and this instance can be saved to the cache pool. Since the instance in the cache pool is in a program-forced static state, caching this instance eliminates the need to maintain its liveness by constructing RTT packets. This cached first instance facilitates its use during the subsequent activation phase.

[0076] In addition, to conserve client hardware resources, a preset lifespan can be established. Once the first instance exceeds this preset lifespan, it can be destroyed to prevent it from consuming excessive resources.

[0077] Step 307: If media transmission with the server is required, media transmission is performed based on the first cached instance and the server.

[0078] Step 307 is the activation phase, corresponding to the stage where the client needs to perform media transmission. This requires media transmission with the server, which can be initiated by the user. For example, the user may use a service requiring media transmission through the client, such as AI dialogue, cloud rendering, or remote customer service. In other words, media transmission with the server is determined when the user needs to engage in AI dialogue or use cloud rendering functionality.

[0079] Regarding the process of media transmission based on the cached first instance and the server, firstly, for the multimedia data involved in the media transmission process, in an optional implementation, as mentioned above, if no real media source is used when creating the first instance, then in step 307, permissions for the media acquisition device can be requested from the operating system. For example, permissions for the camera and / or microphone can be requested by calling the system's API. Then, the fake media source in the first instance is replaced with the media source obtained from the media acquisition device. In this way, the required multimedia data can be transmitted to the server.

[0080] Furthermore, in an optional implementation, during step 307, a third message may be sent to the server. This third message includes the client's configuration information, that is, information that the client needs to send to the server during the connection negotiation process. In other words, the third message includes information that the server needs to obtain from the client during the connection establishment process. This allows the server to obtain the information required for connection negotiation based on the third message and quickly complete the connection establishment with the client.

[0081] In other words, step 307 includes: sending a third message to the server based on the information included in the first message, so that the server can obtain the information required for connection negotiation based on the third message and enable the client to establish a connection with the server by handshaking; and performing media transmission with the server based on the first instance and the connection establishment result.

[0082] Since the connection was established unilaterally by the client in the preceding steps, and the server only sent configuration information to the client, the server did not complete the connection establishment process. Therefore, in step 307, similar to the server sending configuration information, the client can send a third message to the server so that the server can obtain the information needed for the connection establishment process and quickly establish a connection based on the third message. The way the server establishes a connection based on the third message is similar to the way the client establishes a connection in step 303 above, and will not be described in detail here.

[0083] Through the above steps, a two-stage connection establishment model is constructed. In the preparation stage, resources are prepared in advance, all non-sensitive initialization work is completed before the business is triggered, and media collection and data transmission are carried out in the activation stage. The above process separates resource preparation and media activation.

[0084] During the preparation phase, peer connection factories and peer connections are created in advance, saving 70-200ms. Furthermore, the configuration information required for connection negotiation during the connection negotiation process is obtained from the server in advance, and connection negotiation is performed in advance during the preparation phase before the activation phase. The messages sent by the server are simulated based on the configuration information. This process not only reduces latency in the activation phase through pre-negotiation but also reduces latency caused by interaction between the two ends during the preparation phase by constructing the server's messages.

[0085] Furthermore, a permission-free design is adopted during the preparation phase. Instead of activating real media capture devices to collect media sources, track binding is completed using only virtual media sources. This eliminates the need for actual media capture and permission requests for media capture devices during the preparation phase. By postponing permission requests for media capture devices until the user actively interacts, privacy compliance is improved, and the user experience is enhanced.

[0086] During the activation phase, only permission requests for the media capture device and the sending of third-party messages are performed, saving time. Furthermore, regarding the issue of slow media capture device startup, the preparation phase utilizes fake media sources to bind tracks in advance, ensuring that adding tracks during the activation phase does not cause delays.

[0087] Furthermore, during the activation phase, by sending a third message to the server, the server can quickly establish a connection with the client.

[0088] During the activation phase, the above method only requires starting the actual media acquisition device and establishing a network connection, with a total connection establishment time of less than 100ms, thus achieving a user experience of immediate usability upon activation. Furthermore, the method is compatible with the standard WebRTC protocol stack, eliminating the need for a self-developed connection establishment protocol. This method can also simultaneously cache multiple first instances required by various application business scenarios, improving the user experience of multiple services. For example, if the application uses AI dialogue and remote customer service functions, then two first instances corresponding to the two functions can be cached separately.

[0089] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of the apparatus and the computer equipment to which it is applied.

[0090] This specification also provides a media transmission device for use on a client, including:

[0091] The configuration information acquisition module is used to obtain configuration information from the server; the configuration information is the information included in the message returned by the server when the client and the server negotiate a connection.

[0092] The connection negotiation module is used to perform connection negotiation, intercept the first message sent to the server, and construct the second message returned by the server in response to the first message according to the configuration information to complete the connection negotiation.

[0093] The instance caching module is used to build and cache the first instance based on the connection negotiation result;

[0094] The activation module is used to perform media transmission based on the cached first instance and the server when media transmission with the server is required.

[0095] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0096] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0097] like Figure 4 As shown, Figure 4A hardware structure diagram of a computer device is shown. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are internally connected to each other via the bus 1050.

[0098] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above-described methods by running executable instructions.

[0099] The memory 1020 for storing processor-executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020.

[0100] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0101] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0102] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0103] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0104] This specification also provides a computer program product that, when executed by a processor, implements the media transmission method described above.

[0105] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the media transmission method described above.

[0106] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0107] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0109] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A media transmission method, applied to a client, comprising: Obtain configuration information from the server; The configuration information refers to the information included in the messages returned by the server that the client needs to receive during connection negotiation with the server. The connection negotiation is performed by intercepting the first message sent to the server and constructing the second message returned by the server in response to the first message based on the configuration information, so as to complete the connection negotiation. Based on the connection negotiation results, construct and cache the first instance; In cases where media transmission with the server is required, media transmission is performed based on the first cached instance and the server.

2. The method according to claim 1, wherein the connection negotiation includes Session Description Protocol (SDP) negotiation and Interactive Connection Establishment Mechanism (ICE) negotiation.

3. The method according to claim 1, wherein the connection negotiation includes Session Description Protocol (SDP) negotiation and Interactive Connection Establishment (ICE) negotiation; The interception of the first message sent to the server, and the construction of a second message returned by the server in response to the first message based on the configuration information, including: Intercept SDP offer messages sent to the server, and construct an SDP response message returned by the server in response to the SDP offer based on the configuration information; The first ICE message is generated based on the triggering of the SDP offer message; Intercept the first ICE message sent to the server, and construct the second ICE message returned by the server in response to the first ICE message.

4. The method according to claim 2 or 3, further comprising, after completing the connection negotiation: Perform a Datagram Transport Layer Security (DTLS) handshake, intercept the fourth handshake message sent to the server, and construct the fifth handshake message returned by the server based on the fourth handshake message to complete the DTLS handshake. Based on the DTLS handshake result, the connection status is changed to connected.

5. The method according to claim 1, further comprising: By using fake media sources to bind media tracks, the real media source can be simulated using fake media sources in the first instance.

6. The method according to claim 5, wherein the media transmission based on the first instance of the cache and the server comprises: Request permission from the operating system for the media capture device; The fake media source in the first example is replaced with the media source obtained by the media acquisition device.

7. The method according to claim 1, wherein the media transmission based on the first instance of the cache and the server comprises: Based on the information included in the first message, a third message is sent to the server so that the server can obtain the information required for connection negotiation based on the third message and enable the client and the server to shake hands and establish a connection. Based on the first instance and the connection establishment result, media transmission is performed with the server.

8. The method of claim 1, further comprising, before performing connection negotiation: Create a peer-to-peer connection factory; Create peer connections based on the peer connection factory; The connection negotiation includes: Connection negotiation is conducted based on peer connections.

9. The method according to claim 1, wherein intercepting the first message sent to the server and constructing a second message returned by the server in response to the first message based on the configuration information includes: Using a hook, the first message sent to the server is intercepted, and a second message returned by the server in response to the first message is constructed based on the configuration information.

10. The method according to claim 1, wherein the server is an SFU server; The step of obtaining configuration information from the server includes: After the target application starts, obtain configuration information from the server; The method further includes: When users need to engage in AI dialogue or use cloud rendering functions, it is determined that media transmission with the server is required.

11. A computer device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-10 by executing the executable instructions.

Images