Method and apparatus for transmitting multimedia resource, electronic device and storage medium

Abstract

The application discloses a multimedia resource transmission method and device, electronic equipment and storage medium, and belongs to the multimedia technical field. The method comprises the following steps: in response to the role change of a first device in a voice call process being changed into a main speaker, determining a second device in at least one terminal device participating in the same voice call as the first device, wherein the role of the first device in the voice call process is related to the device state of the first device; creating a direct link between the first device and the second device; sending the audio resource of the first device to the second device through the direct link; and sending the audio resource of the first device to a server, so that the server forwards the audio resource of the first device to the second device. The application not only realizes the dynamic creation of the direct link, transmits the multimedia resource through the direct link and the server, but also improves the transmission quality of the multimedia resource while reducing the transmission delay and the packet loss rate of the multimedia resource.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a method, apparatus, electronic device, and storage medium for transmitting multimedia resources. Background Technology

[0002] With the continuous development of terminals and the Internet, more and more people are choosing to use communication software for communication. Common communication software includes instant messaging software and conferencing software. For this type of software, multimedia resources can be transmitted between devices participating in a voice call in the form of data streams.

[0003] In related technologies, multimedia resources include audio resources. Any device participating in a voice call can forward audio resources to other devices participating in the call through a relay server. That is, any device first sends audio resources to the relay server, which then forwards those audio resources to other devices. This method of transmitting audio resources via a relay link suffers from significant transmission latency and packet loss, affecting the transmission quality of the audio resources. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for transmitting multimedia resources, which can be used to solve problems in related technologies. The technical solution includes the following contents.

[0005] Firstly, a method for transmitting multimedia resources is provided, the method comprising:

[0006] In response to the first device changing its role to speaker during a voice call, a second device is determined among at least one terminal device participating in the same voice call as the first device, wherein the role of the first device during the voice call is related to the device state of the first device;

[0007] Create a direct link between the first device and the second device;

[0008] The audio resources of the first device are sent to the second device via the direct link;

[0009] The audio resources of the first device are sent to the server so that the server forwards the audio resources of the first device to the second device.

[0010] Secondly, a method for transmitting multimedia resources is provided, the method comprising:

[0011] In response to receiving a direct connection request sent by the first device forwarded by the server, a direct connection link is created between the second device and the first device. The second device is determined from at least one terminal device participating in the same voice call as the first device when the role of the first device changes to speaker during the voice call. The role of the first device during the voice call is related to the device state of the first device.

[0012] The audio resources of the first device are received via the direct link;

[0013] Receive the audio resources of the first device forwarded by the server.

[0014] Thirdly, a multimedia resource transmission device is provided, the device comprising:

[0015] The determination module is used to determine a second device among at least one terminal device participating in the same voice call as the first device in response to the first device changing its role to speaker during the voice call. The role of the first device during the voice call is related to the device state of the first device.

[0016] A creation module is used to create a direct link between the first device and the second device;

[0017] The sending module is used to send the audio resources of the first device to the second device via the direct connection link;

[0018] The sending module is further configured to send the audio resources of the first device to the server, so that the server forwards the audio resources of the first device to the second device.

[0019] Fourthly, a multimedia resource transmission device is provided, the device comprising:

[0020] A creation module is used to respond to a direct connection request sent by a first device forwarded by a server and create a direct connection link between a second device and the first device. The second device is determined from at least one terminal device participating in the same voice call as the first device when the role of the first device changes to speaker during a voice call. The role of the first device during the voice call is related to the device state of the first device.

[0021] The receiving module is used to receive audio resources of the first device sent by the first device through the direct connection link;

[0022] The receiving module is also used to receive the audio resources of the first device forwarded by the server.

[0023] Fifthly, a multimedia resource transmission system is provided, characterized in that the system includes a first device, a second device, and a server;

[0024] The first device is used to perform the method for transmitting multimedia resources as described in the first aspect;

[0025] The second device is used to perform the method for transmitting multimedia resources as described in the second aspect;

[0026] The server is used to perform the functions performed by the server as described in the first or second aspect.

[0027] In a sixth aspect, an electronic device is provided, the electronic device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to implement any of the multimedia resource transmission methods described above.

[0028] In a seventh aspect, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for transmitting multimedia resources.

[0029] Eighthly, a computer program or computer program product is also provided, wherein the computer program or computer program product stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for transmitting multimedia resources.

[0030] The technical solution provided in this application brings at least the following beneficial effects:

[0031] In the technical solution provided in this application, when the first device changes its role to speaker during a voice call, a second device is identified from at least one terminal device participating in the same voice call as the first device, and a direct link is created between the first device and the second device, thus achieving dynamic creation of the direct link. On the one hand, the audio resources of the first device are sent to the second device through the direct link to reduce the transmission latency and packet loss rate of the audio resources. On the other hand, the audio resources of the first device are sent to the server so that the server can forward the audio resources of the first device to the second device, thereby improving the transmission quality of the audio resources. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram illustrating the implementation environment of a multimedia resource transmission method provided in this application embodiment;

[0034] Figure 2 This is a flowchart illustrating a method for transmitting multimedia resources according to an embodiment of this application;

[0035] Figure 3 This is a flowchart of another method for transmitting multimedia resources provided in an embodiment of this application;

[0036] Figure 4 This is a flowchart of another multimedia resource transmission method provided in the embodiments of this application;

[0037] Figure 5 This is a schematic diagram illustrating a role switching method provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram illustrating the creation and closure of a direct link according to an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of the transmission of multimedia resources provided in an embodiment of this application;

[0040] Figure 8 This is a schematic diagram of the structure of a multimedia resource transmission device provided in an embodiment of this application;

[0041] Figure 9 This is a schematic diagram of another multimedia resource transmission device provided in an embodiment of this application;

[0042] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0043] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] Figure 1 This is a schematic diagram illustrating the implementation environment of a multimedia resource transmission method provided in this application embodiment, such as... Figure 1 As shown, the implementation environment includes at least two terminal devices and a server (the server includes the relay server mentioned below). When various terminal devices participate in the same voice call through the server, the multimedia resource transmission method in this embodiment can be executed by any of the terminal devices.

[0046] Each terminal device has a direct link established with the server. The relay link between two terminal devices includes a direct link between one terminal device and the server, and a direct link between the other terminal device and the server. In other words, transmitting multimedia resources between two terminal devices via a relay link means that one terminal device sends multimedia resources to the server through its direct link, and the server forwards the multimedia resources to the other terminal device through its direct link.

[0047] In this embodiment of the application, any terminal device can send multimedia resources to the server, and the server can forward the multimedia resources to other terminal devices, thereby enabling any terminal device to forward multimedia resources to other terminal devices through a relay link. Figure 1 The diagram shows that terminal device 1 can send multimedia resources to the server, and the server can forward the multimedia resources to terminal devices 2 and 3, thereby enabling terminal device 1 to forward multimedia resources to terminal devices 2 and 3 through a relay link.

[0048] Furthermore, any terminal device can establish a direct link with other terminal devices, allowing any terminal device to directly send multimedia resources to other terminal devices. For example... Figure 1 The diagram shows that terminal device 1 has established direct links with terminal devices 2 and 3 respectively, enabling terminal device 1 to directly send multimedia resources to terminal devices 2 and 3.

[0049] Terminal devices can be smartphones, game consoles, desktop computers, tablets, laptops, smart TVs, smart in-vehicle devices, smart voice interaction devices, smart home appliances, etc. The server can be a single server, a server cluster consisting of multiple servers, or any of the following: a cloud computing platform or a virtualization center; this application embodiment does not limit this. The server can communicate with the terminal devices via a wired or wireless network. The server can have functions such as data processing, data storage, and data transmission and reception; this application embodiment does not limit this. The number of terminal devices and servers is not limited; there can be one or more.

[0050] To facilitate understanding of the embodiments of this application, the terms that may be involved in the embodiments of this application will be explained and described below.

[0051] Relay Link: Also known as a Selective Forwarding Unit (SFU) relay link, an SFU is a transmission architecture consisting of a relay server and multiple terminal devices in a star topology. Each terminal device sends its shared multimedia resources to the relay server, which then forwards these resources to other terminal devices. During a voice call, when two terminal devices transmit multimedia resources via a relay link, the data needs to be forwarded by the relay server. That is, one terminal device can send multimedia resources to another through a relay link, and can also receive multimedia resources sent by another terminal device through a relay link. Relay links have high establishment success rates and availability, but also higher transmission costs. In this embodiment, the relay link is a basic transmission link.

[0052] Direct Link: A direct link, also known as a mesh network direct link, is a network structure formed by connecting multiple terminal devices in pairs. For example, three terminal devices, A, B, and C, are connected in pairs. When A wants to share multimedia resources (such as audio and video), A sends the multimedia resources to B and C respectively. Similarly, when B wants to share multimedia resources, it needs to send the multimedia resources to A and C respectively, and so on. The mesh network architecture requires each terminal device to establish direct links with all other terminal devices, which is complex and difficult. During a voice call, when two terminal devices transmit multimedia resources through a direct link, they do not need to go through a relay server (there may be network nodes such as routers and switches forwarding). Since the direct link does not go through a relay server, the transmission cost is lower and it can be used to transmit large amounts of data (such as audio and video resources). In the embodiments of this application, the direct link between two terminal devices is dynamically created and closed based on the role of the terminal device in the voice call process. Therefore, the direct link is an auxiliary transmission link. When relay links are congested, using direct links to transmit multimedia resources can effectively improve transmission quality.

[0053] Network Address Translation (NAT) is a technique that rewrites at least one of the source IP address, source port address, destination IP address, and destination port address when Internet Protocol (IP) packets pass through routers or firewalls. NAT can solve the problem of insufficient Internet Protocol version 4 (IPv4) addresses and effectively prevent attacks from outside the network, hiding and protecting computers inside the network. Depending on the mapping and filtering rules, NAT is divided into full-cone, restricted, port-restricted, and symmetric types. Among these, port-restricted and symmetric NAT offer the highest security levels, have the most stringent connectivity requirements, and are the most widely used.

[0054] NAT traversal, also known as NAT hole punching, addresses the issue of NAT mapping. NAT maps internal IPv4 addresses to external addresses. Furthermore, NAT filters incoming packets from the external network according to certain rules. This complicates communication between two NAT-connected internal hosts. NAT traversal technology is used to break down the NAT barrier and establish a direct link between two NAT-connected internal hosts.

[0055] In the field of multimedia technology, communication software is a common type of software. At least two terminal devices with communication software installed can participate in the same voice call. During a voice call involving at least two terminal devices, any one terminal device can capture multimedia resources such as sound and images and send these resources to the other terminal devices participating in the voice call.

[0056] In related technologies, any terminal device sends multimedia resources to a relay server, which then forwards these resources to other terminal devices, thus enabling any terminal device to send multimedia resources to other terminal devices via a relay link. However, because the multimedia resources need to be forwarded through the relay server, the above technology suffers from significant transmission latency and packet loss, resulting in poor transmission quality of the multimedia resources.

[0057] This application provides a method for transmitting multimedia resources, which can be used to solve the above-mentioned problems. Furthermore, the method provided in this application can be applied to the above-mentioned implementation environment. Figure 2 The flowchart shown in this embodiment of the application illustrates a method for transmitting multimedia resources. This method can be implemented by... Figure 1The method is executed by any terminal device (such as terminal device 1). Since the above embodiment involves at least two terminal devices, for ease of description, the terminal device executing the multimedia resource transmission method provided in this application embodiment is referred to as the first device, and the terminal devices other than the first device among the at least two terminal devices are referred to as other terminal devices. Figure 2 As shown, the method includes at least steps 201 to 204 as shown below.

[0058] Step 201: In response to the first device changing its role to speaker during a voice call, a second device is determined among at least one terminal device participating in the same voice call as the first device. The role of the first device during the voice call is related to the device state of the first device.

[0059] In this embodiment, at least two terminal devices participate in the same voice call. These at least two terminal devices can be divided into a first device and other terminal devices, where the other terminal devices are equivalent to "at least one terminal device participating in the same voice call with the first device" in step 201. The first device and the other terminal devices have the same communication software installed, through which they can participate in the same voice call. For example, when the communication software is instant messaging software, the first device and the other terminal devices can participate in a two-person voice call. When the communication software is conferencing software, the first device and the other terminal devices can participate in an audio-visual conference. In an audio-visual conference, two or more individuals or groups in different locations can exchange resources such as sound, images, and files through the transmission link between the terminal devices, thereby achieving real-time interactive communication.

[0060] This application employs an adaptive role-switching strategy to assign roles to terminal devices participating in a voice call. The role of a terminal device during a voice call is related to its device state. Taking a terminal device as the first device as an example, as the device state changes, the first device's role in the voice call may change to that of a speaker. A speaker is a role of a terminal device during a voice call, responsible for outputting multimedia resources. These multimedia resources include, but are not limited to, at least one of text, images, audio, video, and files. When multimedia resources include both audio and video resources, they can be referred to as audio-visual resources. Multiple speakers may exist simultaneously during a voice call.

[0061] The first device's role changes to that of the speaker during a voice call, and there are at least two scenarios as shown below: A1 and A2.

[0062] In case A1, before step 201, the method further includes: in response to the device state of the first device changing to the video resource acquisition state, determining that the role of the first device in the voice call process is directly changed to the speaker.

[0063] When the first device enables video resource acquisition, such as by enabling screen sharing (in which case the first device can capture information from the screen) or opening its camera (in which case the first device can capture information captured by the camera), the device status of the first device changes from a state of disabled video resource acquisition to a state of video resource acquisition. The server can detect this change in real time (i.e., information indicating that the first device's device status has changed from a state of disabled video resource acquisition to a state of video resource acquisition). Alternatively, the first device can send this change information to the server, which, based on this change information, determines that the first device's role in the voice call has been directly changed to speaker, and sends the relevant information regarding the direct change of the first device's role to speaker in the voice call to the first device. For ease of description, the role of the first device in the voice call will be referred to as the role of the first device in the following description.

[0064] In case A2, before step 201, the method further includes: in response to the device state of the first device changing to the audio resource acquisition state, acquiring the audio resources acquired by the first device; in response to detecting the object audio in the audio resources acquired by the first device, determining that the role of the first device changes to the speaker during the voice call.

[0065] When the first device enables audio resource acquisition—for example, when the first device turns on its microphone (at this time, the first device can acquire the subject's speech or music played by a music player), the device status of the first device changes from audio resource acquisition disabled to audio resource acquisition enabled. The server can detect this change in real time, or the first device can send this change information to the server. Based on this change information, the server determines that the first device's role changes to an intermediate state and sends relevant information about this change to the first device. The intermediate state is a role of the terminal device during a voice call; it can change to speaker or revert to listener.

[0066] When the first device changes its role to an intermediate state, it can acquire audio resources in this intermediate state. One possible implementation is that the first device performs Voice Activity Detection (VAD) on the audio resource, obtains the detection result, and sends it to the server. If the detection result indicates the presence of object audio in the audio resource, the server determines that the first device's role has changed from the intermediate state to that of the speaker and feeds this role change information back to the first device. If the detection result indicates the absence of object audio in the audio resource, the server determines that the first device's role has changed from the intermediate state to that of the listener and feeds this role change information back to the first device. Here, the listener is a role of the terminal device during a voice call, responsible for receiving and playing the multimedia resources output by the speaker. During a voice call, the listener can become the speaker, and similarly, the speaker can become the listener.

[0067] Another possible implementation involves the first device sending the audio resource to a server, which then performs voice activity detection on the audio resource. When the server detects the presence of target audio in the audio resource, it determines that the first device's role changes from an intermediate state to that of the speaker, and this role change information is fed back to the first device. When the server detects that the target audio is not present in the audio resource, it determines that the first device's role changes from an intermediate state to that of the listener, and this role change information is fed back to the first device.

[0068] In this embodiment of the application, when the device state of the first device changes to the audio resource acquisition state, the role of the first device is first determined to change to an intermediate state, and then the audio resources acquired by the first device in the intermediate state are detected. Only after the object's audio is detected is the role of the first device determined to change to the speaker. This prevents the first device from changing to the speaker when the object turns on the microphone (or accidentally turns on the microphone) and does not speak. This can avoid establishing a direct link between the first device and other terminal devices to a certain extent.

[0069] It should be noted that before the role of the first device changes to speaker, the first device's role can be listener, no role, or an intermediate state. Before the first device's role changes to an intermediate state, the first device's role can be listener or no role. The above describes the situation where the first device's role changes to speaker; the first device's role can first change to an intermediate state, and then change to speaker or listener. Below are brief descriptions of other cases where the first device's role changes to listener (denoted as cases A3 and A4), and the case where the first device's role changes to no role (denoted as case A5).

[0070] In scenario A3, when the device status of the first device changes from not participating in a voice call to participating in a voice call, the role of the first device is determined to change from having no role to being a listener.

[0071] The first device can create and participate in a voice call. In this case, the device state changes from "not participating in a voice call" to "participating in a voice call." Alternatively, if the server is involved in the voice call, the first device, which is not currently participating, can send a request to the server to participate in the call. When the first device receives a response from the server, it participates in the voice call, changing its state from "not participating in a voice call" to "participating in a voice call." Or, if other terminal devices are involved in the voice call, these devices can send an invitation to the first device to participate. When the first device responds to the invitation, it participates in the voice call, changing its state from "not participating in a voice call" to "participating in a voice call."

[0072] When the device status of the first device changes from not participating in a voice call to participating in a voice call, the server can detect this change information in real time. Alternatively, the first device can send this change information to the server, and the server can determine the first device's role in the voice call process from no role to listener based on this change information, and send the relevant information of the first device's role change to listener to the first device.

[0073] In scenario A4, when the device status of the first device changes from multimedia resource acquisition to multimedia resource acquisition disabled, the role of the first device is determined to change to listener. The multimedia resource acquisition status here includes audio resource acquisition and / or video resource acquisition. The multimedia resource acquisition disabled status here includes both audio and video resource acquisition being disabled.

[0074] When the device status of the first device changes from multimedia resource acquisition to multimedia resource acquisition disabled, the server can detect this change in real time, or the first device can send this change information to the server. If, for a period of time (e.g., time T is 1 minute), the server obtains that the device status of the first device remains in the multimedia resource acquisition disabled state, the server determines that the role of the first device has changed from speaker or intermediary to listener, and sends the relevant information about the change in the first device's role to listener to the first device.

[0075] In case A5, when the device status of the first device changes from participating in a voice call to not participating in a voice call, the role of the first device is determined to have changed to no role.

[0076] When the first device requests to leave a voice call, its device status changes from participating in a voice call to not participating in a voice call. The server can detect this change in real time, or the first device can send this change information to the server. At this time, the server determines that the first device's role has changed from speaker, intermediary, or listener to no role, and sends the relevant information about the first device's role change to no role to the first device. When the first device receives the notification that its role has changed to no role, the first device leaves the voice call.

[0077] It is understandable that when the first device is in a state of not participating in a voice call, since the first device is not participating in a voice call, it cannot play any role during the voice call. At this time, the role of the first device is no role.

[0078] It should be noted that when the first device is the speaker in a voice call, it is the producer of multimedia resources, primarily responsible for acquiring, encoding, and transmitting the encoded multimedia resources to other terminal devices. Furthermore, the first device is also a receiver of multimedia resources, capable of receiving multimedia resources sent by the terminal device in the speaker's role, decoding them, and playing the decoded multimedia resources.

[0079] When the first device acts as a listener during a voice call, it is the receiver of multimedia resources, primarily used to receive multimedia resources sent by the terminal device acting as the speaker. Since multimedia resources typically require encoding before transmission, the terminal device, upon receiving the multimedia resources, can first decode them before playing them.

[0080] When the first device is in an intermediate state during a voice call, it can receive multimedia resources sent by the terminal device that is the speaker, decode the multimedia resources, and play the decoded multimedia resources. On the other hand, since the first device has audio capture enabled, it can capture audio resources, but these audio resources will not be sent to the terminal device that is the speaker or the listener.

[0081] Depending on the application scenario, the number of terminal devices corresponding to the speaker and listener in a voice call varies. For example, in a two-person voice call scenario, generally two terminal devices are involved. Either of these two terminal devices can act as the speaker or the listener, and the speaker and listener roles of each terminal device can be switched in real time. That is to say, any terminal device that is acting as the speaker one moment may act as the listener the next.

[0082] In a remote conferencing scenario, there may be several or even a dozen terminal devices involved. Some of these terminal devices may always be in the role of listeners, while others can switch from speaker to listener or vice versa in real time.

[0083] In online classrooms, there may be dozens or even hundreds of terminal devices. Most of these devices may act as listeners, while a few may act as speakers. For example, the teacher's device may act as the speaker, while the student's device may act as a listener.

[0084] In this embodiment, when the first device changes its role to speaker during a voice call, a second device is determined among the other terminal devices. There is at least one second device, and its role can be speaker, listener, or an intermediary. The principle of the second device's role change is similar to that of the first device, as described above in the relevant descriptions of the first device's role change (e.g., cases A1 to A5), and will not be repeated here.

[0085] Step 202: Create a direct link between the first device and the second device.

[0086] It should be noted that for any second device, if a direct link has already been established between the second device and the first device, the first device does not need to establish a direct link with the second device again. That is, for the second device, the first device can skip step 202 and directly execute steps 203 and 204. If the second device has not established a direct link with the first device, the first device needs to establish a direct link with the second device. That is, for the second device, the first device needs to execute step 202.

[0087] In this embodiment, when the device state of the first device changes to a state of participating in a voice call, the first device creates a direct-connection User Datagram Protocol Socket (UDPSocket), which can be simply referred to as a direct-connection Socket. A direct-connection Socket is a data structure used to store the state information of all direct-connection links in the first device. These direct-connection links include the direct-connection link between the first device and a server (the server includes at least one of a direct-connection burrowing server and a relay server, wherein the direct-connection burrowing server can determine the external network address of the terminal device and return it to the terminal device). When resource transmission occurs between the first device and the server, the direct-connection link between the first device and the server needs to be determined based on the direct-connection Socket, and resource transmission is performed based on the determined direct-connection link.

[0088] Similarly, when the second device's device state changes to participate in a voice call, the second device will also create a direct connection socket. After the first and second devices have created their respective direct connection sockets, a direct connection link between the first and second devices is created based on their respective direct connection sockets.

[0089] In one possible implementation, creating a direct link between the first device and the second device includes: sending a direct link request to the server for the second device, the direct link request carrying the external network address of the first device; receiving a direct link response forwarded by the server, the direct link response being the response information sent by the second device to the server in response to the direct link request after receiving the forwarded direct link request, the direct link response carrying the external network address of the second device; extracting the external network address of the second device from the direct link response; and creating a direct link between the first device and the second device based on the external network addresses of the first device and the second device.

[0090] The first device, based on its direct connection socket, determines the direct link between itself and the direct connection punching server, and sends UDP packets to the server through this link. Optionally, the first device generates a raw UDP packet and sends it to the router. The router fills the first device's external IP address into the raw UDP packet to obtain the target UDP packet, and then sends the target UDP packet to the direct connection punching server. The first device's external IP address includes its Internet Protocol address (IP address) and port address. The IP address is used to locate the first device, and the port address is used to locate its application.

[0091] After receiving the target UDP packet, the direct-connect UDP server parses the packet to extract the external IP address of the first device, and then sends this address back to the first device. Thus, the first device can obtain and store its own external IP address. Based on the same principle, the second device can also obtain and store its own external IP address.

[0092] The first device can send a direct connection request to the relay server for the second device. This request includes the first device's external IP address, the identification information of the first device (as the source device), and the identification information of the second device (as the destination device). Upon receiving the direct connection request, the relay server parses it to obtain the second device's identification information and forwards the request to the second device based on that information. It should be noted that the direct connection request forwarded by the relay server to the second device includes at least the first device's identification information and its external IP address; it may or may not include the second device's identification information.

[0093] After receiving a direct connection request, the second device parses the request to obtain the identification information and external network address of the first device, and stores these two information. Additionally, the second device sends a direct connection response to the relay server, which includes the external network address of the second device, the identification information of the first device (as the target device), and the identification information of the second device (as the source device). Upon receiving this direct connection response, the relay server parses it to obtain the identification information of the first device and forwards the direct connection request to the first device based on this information. It should be noted that the direct connection response forwarded by the relay server to the first device includes at least the identification information and the external network address of the second device, but may or may not include the identification information of the first device.

[0094] After receiving the direct connection response, the first device parses the response to obtain the identification information and external network address of the second device, and stores the identification information and external network address of the second device. From then on, both the first and second devices can obtain and store each other's external network address.

[0095] For example, a first device m sends a direct connection request to a relay server for a second device n. This request carries (IP_m, port_m), id(m), and id(n). (IP_m, port_m) are the external network addresses of the first device m, where IP_m is the IP address of the first device m, port_m is the port address of the first device m, id(m) is the identification information of the first device m, and id(n) is the identification information of the first device n. After the relay server forwards this direct connection request to the second device n, the second device n parses and stores id(m) and (IP_m, port_m), and then sends a direct connection response to the relay server for the first device m. This response carries (IP_n, port_n), id(m), and id(n). (IP_n, port_n) are the external network addresses of the second device n, where IP_n is the IP address of the second device n, and port_n is the port address of the second device n. After the relay server forwards the direct connection response to the first device m, the first device m parses out id(n) and (IP_n, port_n) and stores them.

[0096] After the first and second devices obtain each other's external network addresses, it is equivalent to creating a direct link between the first and second devices.

[0097] Optionally, based on the external network address of the first device and the external network address of the second device, a direct link between the first device and the second device is created, including: sending multiple direct link data packets to the second device based on the external network address of the second device; and successfully creating a direct link between the first device and the second device in response to receiving a reception response from the second device based on the external network address of the first device for each direct link data packet.

[0098] The first device can send multiple direct connection data packets to the second device based on the second device's external network address. These direct connection data packets are UDP packets as mentioned above. Each time the second device receives one (or more) direct connection data packets, it sends a reception response to the first device based on the first device's external network address for that direct connection data packet (or the multiple direct connection data packets). When the first device receives the reception responses from the second device for each direct connection data packet, the first device determines that a direct connection link between the first and second devices has been successfully established. In other words, the first device has confirmed that the second device's external network address is correct, and the second device has successfully received the first device's external network address.

[0099] Based on the same principle as the first device, the second device can send multiple direct connection data packets to the first device using the first device's external network address. Each time the first device receives one (or more) direct connection data packets, it sends a response to the second device based on the second device's external network address, acknowledging the direct connection. When the second device receives the response from the first device for each direct connection data packet, it determines that a direct connection link has been successfully established between the second and first devices. In other words, the second device has confirmed that the first device's external network address is correct, and the first device has successfully received the second device's external network address.

[0100] The method by which the first and second devices have each successfully established a direct link between them is also known as direct link burr punching. Direct link burr punching tests whether a direct link has been successfully established between the first and second devices and verifies the stability of the direct link. This facilitates improving the success rate of multimedia resource transmission when transmitting multimedia resources via the direct link in the future.

[0101] Step 203: Send the audio resources of the first device to the second device via a direct link.

[0102] In this embodiment, when the first device acts as the speaker, it can acquire its own audio resources and transmit them to the second devices via a direct link. There is at least one second device, and the first device transmits its audio resources to each of the second devices via a direct link. This direct link transmission allows each second device that has established a direct link with the first device to quickly and accurately obtain its audio resources, effectively utilizing the direct link to reduce transmission latency and packet loss.

[0103] Step 204: Send the audio resources of the first device to the server, so that the server forwards the audio resources of the first device to the second device. Wherein, Figure 2 Step 204 is shown to be executed after step 203. Optionally, step 204 may also be executed after step 201 or step 202, which is not limited here.

[0104] When the first device acquires the audio resources of the first device, it sends the audio resources of the first device to the server so that the server can forward the audio resources of the first device to the second device. Through steps 203 and 204, the second device can receive the audio resources of the first device not only through a direct link, but also through a relay link. This reduces the possibility that the second device may not be able to receive the audio resources of the first device due to congestion on a certain link, and improves the transmission quality of the audio resources.

[0105] In this embodiment of the application, the server can forward the audio resources of the first device to the second device within a set time period, that is, the server unconditionally forwards the audio resources of the first device to the second device within the set time period.

[0106] After a set time period, the server conditionally forwards the audio resources of the first device to the second device. Optionally, the server forwards the audio resources of the first device to the second device when the first forwarding condition is met; the server stops forwarding the audio resources of the first device to the second device when the first forwarding condition is not met.

[0107] In this embodiment, the server determines whether the second device meets the first forwarding condition. If the second device meets the first forwarding condition, the server forwards the audio resources of the first device to the second device. If the second device does not meet the first forwarding condition, the server stops forwarding the audio resources of the first device to the second device.

[0108] Optionally, satisfying the first forwarding condition includes the second device's relay data packet utilization rate being greater than a first utilization threshold. The second device's relay data packet utilization rate is used to characterize the proportion of data packets forwarded through the server used by the second device, that is, the ratio between the number of data packets forwarded through the server used by the second device and the total number of data packets used by the second device. The total number of data packets used by the second device is equal to the sum of the number of data packets forwarded through the server and the number of data packets sent through the direct link by the second device.

[0109] The first device sends its audio resources to the server. After receiving the audio resources from the first device, if the utilization rate of the relay data packets from the second device to the first device is greater than a first utilization threshold, the server forwards the audio resources from the first device to the second device.

[0110] It should be noted that the audio resources of the first device are continuously sent to the server and the second device in the form of a stream, and the server also continuously forwards the audio resources of the first device to the second device in the form of a stream. When the audio resources of the first device are transmitted in the form of a stream, the audio resources of the first device correspond to the data packets of the first device.

[0111] On one hand, the second device continuously receives data packets sent by the first device via a direct link. On the other hand, the second device continuously receives data packets forwarded by a relay server via a relay link. Each time the second device receives a data packet, it obtains the packet's sequence number. If the second device has already received the data packet corresponding to this sequence number, it discards the packet. If the second device has not received the data packet corresponding to this sequence number, it can use the data packet; that is, it parses the data packet and plays the corresponding audio resource. In this way, the second device deduplicates the data packets received via the direct and relay links. By counting the number of data packets received via the relay link and the number of data packets received via the direct link, the utilization rate of the relayed data packets from the first device can be obtained. The data packets received via the relay link are the same as the data packets forwarded by the relay server.

[0112] Optionally, the relay data packet utilization rate of the second device to the first device is equal to the ratio of the number of data packets received by the second device using the relay link to the total number of data packets used by the second device, wherein the total number of data packets used by the second device is equal to the sum of the number of data packets received by the second device using the relay link and the number of data packets received by the second device using the direct link. That is, the relay data packet utilization rate of the second device to the first device satisfies the following formula (1).

[0113]

[0114] in, The relay data packet utilization rate of the second device j to the first device i is represented by the data packet in formula (1).

[0115] The second device can send the relay data packet utilization rate of the first device to the relay server, so that the relay server can determine whether to forward the data packets of the first device to the second device based on the relay data packet utilization rate.

[0116] Optionally, the relay server can first forward data packets from the first device to the second device, so that the second device can calculate the utilization rate of the relayed data packets for the first device within a set time period and forward this utilization rate to the relay server. Then, if the relay server determines that the utilization rate of the second device's relayed data packets is greater than a first utilization threshold, the server forwards the data packets from the first device to the second device. If the relay server determines that the utilization rate of the second device's relayed data packets is not greater than the first utilization threshold, the server does not forward the data packets from the first device to the second device.

[0117] As described above, the first device sends its audio resources to the second device using a dual-transmission strategy of a relay link and a direct link. When the utilization rate of the relay data packets is no greater than a first utilization threshold, it indicates that the second device is highly likely to use data packets received via the direct link (which also demonstrates the high transmission quality and stability of the direct link) and will likely discard data packets forwarded by the relay server. Therefore, the data packets sent by the relay server are likely to be invalid transmissions. When the utilization rate of the second device's relay data packets is no greater than the first utilization threshold, the relay server stops forwarding data packets from the first device to the second device, reducing invalid transmissions and conserving downlink resources.

[0118] It is understandable that the utilization rate of the relay data packets of the second device is greater than or equal to 0 and less than or equal to 1. The first utilization threshold is an adjustable coefficient, denoted as α, where 0 ≤ α ≤ 1.

[0119] If α = 0, then the relay data packet utilization rate of the second device is greater than or equal to α.

[0120] In this scenario, if the relay data packet utilization rate of the second device is greater than α, the server forwards the data packets from the first device to the second device; if the relay data packet utilization rate of the second device is equal to α, the server may or may not forward the data packets from the first device to the second device. In one possible implementation, regardless of whether the relay data packet utilization rate of the second device is greater than or equal to α, the relay server forwards the audio resources from the first device to the second device.

[0121] If α = 1, then the relay data packet utilization rate of the second device is less than or equal to α.

[0122] In this scenario, if the relay data packet utilization rate of the second device is less than α, the server will not forward the data packets of the first device to the second device; if the relay data packet utilization rate of the second device is equal to α, the server may or may not forward the data packets of the first device to the second device. In one possible implementation, regardless of whether the relay data packet utilization rate of the second device is less than or equal to α, the relay server will not forward the audio resources of the first device to the second device.

[0123] It should be noted that among the other terminal devices participating in the same voice call as the first device, there may be terminal devices that have not established a direct connection with the first device. For these terminal devices, the first device cannot send its audio resources to them via a direct connection, but can only send them via a relay link. Therefore, the utilization rate of the relayed data packets from the first device to these terminal devices is 1, which is greater than the first utilization threshold. Therefore, the relay server will continuously send the first device's data packets to these terminal devices, ensuring that terminal devices that have not established a direct connection with the first device can also receive the first device's audio resources.

[0124] In summary, any terminal device participating in the same voice call as the first device may have established a direct link with the first device. This terminal device could be the second device mentioned above, or it may not have established a direct link with the first device. Let the first device be denoted as i, and any terminal device as j. We can then define whether a direct link has been established between the first device i and the terminal device j according to the formula (2) shown below.

[0125]

[0126] Among them, C ij This can also be called direct reachability, C ij =1 indicates that a direct link has been established between the first device i and the terminal device j, C ij =0 indicates that no direct link has been established between the first device i and the terminal device j.

[0127] For audio resources, a dual-transmission strategy using both direct and relay links is employed. When the first device i sends its audio / video resources to the terminal device j, on the one hand, the first device i will use a relay link to transmit the audio resources to the terminal device j; on the other hand, the first device i will check the direct reachability between itself and the terminal device j. If C ij If C = 1, then the first device i will use a direct link to transmit the audio resource to the terminal device j. ij If the value is 0, then the first device i cannot use the direct link to transmit the audio resource to the terminal device j. By employing a dual-transmission strategy using both direct and relay links, link resources are fully utilized, thereby improving the transmission quality of the audio resource.

[0128] In one possible implementation, please see Figure 3 , Figure 3 This is a flowchart of another multimedia resource transmission method provided in an embodiment of this application, which further includes step 205. Wherein, Figure 3Step 205 is shown to be executed after step 202. Optionally, step 205 may also be executed after step 203 or step 204, which is not limited here.

[0129] Step 205: Obtain the video resources of the first device and determine the video bitrate of the video resources of the first device; when the available uplink bandwidth of the first device is not less than the product between the video bitrate of the video resources of the first device and the reference quantity, send the video resources of the first device to the second device through the direct link.

[0130] In this embodiment of the application, when the first device acts as the speaker, it can acquire its own video resources and determine the video bitrate of those resources. This video bitrate characterizes the number of data bits transmitted per unit time when transmitting the video resources. The video bitrate can also be called the sampling rate, and its unit is typically kilobits per second (kbps) or megabits per second (mbps).

[0131] The first device can calculate the product between the video bitrate of its video resources and a reference number to obtain the product result. The reference number can be a set value, such as the number of terminal devices participating in the voice call, or the number of second devices. In this embodiment, any terminal device with a direct link established with the first device is considered a second device as mentioned above; therefore, the number of second devices is at least one.

[0132] Furthermore, the first device can also determine its available uplink bandwidth. The available uplink bandwidth of the first device is its total uplink bandwidth minus its already utilized uplink bandwidth. The utilized uplink bandwidth includes the uplink bandwidth required for the first device to send audio resources to the relay server, the uplink bandwidth required for the first device to send audio resources to the second device via a direct link, and the uplink bandwidth required for the first device to send video resources to the relay server. Since the uplink bandwidth required to send audio resources is relatively small and can be ignored, the available uplink bandwidth of the first device can be roughly estimated as the difference between its total uplink bandwidth and the uplink bandwidth required for the first device to send video resources to the relay server.

[0133] In this embodiment, if the available uplink bandwidth of the first device is not less than the product result, the first device sends its video resources to the second device through the direct link between the first device and the second device. Here, the fact that the available uplink bandwidth of the first device is not less than the product result indicates that the available uplink bandwidth of the first device is large enough to support sending the first device's video resources to each of the second devices through the direct link. This reduces the transmission latency and packet loss rate of the video resources.

[0134] In one possible implementation, please see Figure 3 The method further includes step 206. Among them, Figure 3 Step 206 is shown to be executed after step 202. Optionally, step 206 may also be executed after step 203 or step 204, which is not limited here.

[0135] Step 206: Obtain the video resources of the first device and determine the video bitrate of the video resources of the first device; when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity, and the second device meets the first transmission condition, send the video resources of the first device to the second device through the direct link.

[0136] In this embodiment, if the available uplink bandwidth of the first device is less than the product result, it indicates that the available uplink bandwidth of the first device is insufficient, and it is not supported to send the video resources of the first device to each of the second devices through direct links. In other words, the first device cannot send the video resources of the first device to all the second devices through direct links. At this time, for each second device, the first device needs to determine whether the second device meets the first transmission condition.

[0137] Optionally, the second device satisfies the first transmission condition, including: the direct connection data packet utilization rate of the second device is not less than a second utilization threshold, whereby the direct connection data packet utilization rate of the second device characterizes the proportion of data packets sent by the second device via the direct connection link. In other words, the direct connection data packet utilization rate of the second device characterizes the ratio between the number of data packets sent by the second device via the direct connection link and the total number of data packets used by the second device. The total number of data packets used by the second device is equal to the sum of the number of data packets forwarded by the server and the number of data packets sent by the second device via the direct connection link.

[0138] As mentioned above, the second device deduplicates the data packets received through the direct link and the relay link. By counting the number of data packets received by the second device through the relay link and the number of data packets received by the second device through the direct link, the utilization rate of the direct data packets of the second device to the first device can be obtained.

[0139] Optionally, the direct data packet utilization rate of the second device to the first device is equal to the ratio of the number of data packets received by the second device using the direct link to the total number of data packets used by the second device, wherein the total number of data packets used by the second device is equal to the sum of the number of data packets received by the second device using the relay link and the number of data packets received by the second device using the direct link. That is, the direct data packet utilization rate of the second device to the first device satisfies the following formula (3).

[0140]

[0141] in, The direct data packet utilization rate of the second device j relative to the first device i is represented by the data packet in formula (3).

[0142] The sum of the direct data packet utilization rate of the second device to the first device and the relay data packet utilization rate of the second device to the first device is 1, that is... Therefore, after determining the utilization rate of direct data packets to the first device, the second device is essentially determining the utilization rate of relay data packets to the first device. Similarly, after determining the utilization rate of relay data packets to the first device, the second device is essentially determining the utilization rate of direct data packets to the first device.

[0143] The second device can send the utilization rate of direct data packets for the first device to the first device via the direct link between the second device and the first device, so that the first device can obtain the utilization rate of the direct data packets for the second device. The first device can determine whether the utilization rate of the direct data packets for the first device by the second device is not less than a second utilization threshold. If it is not less than a second utilization threshold, the second device meets the first transmission condition. At this time, the first device sends its video resources to the second device via the direct link, thereby reducing the transmission latency and packet loss rate of the video resources. Optionally, the second utilization threshold can be based on data from an empirical device, for example, the second utilization threshold is 0.5.

[0144] In one possible implementation, there is at least one second device. Each second device can send its direct data packet utilization rate to the first device via a direct link, allowing the first device to obtain the direct data packet utilization rates of each second device. The direct data packet utilization rates of each second device are sorted, and a second utilization threshold can be determined based on the sorting result. For example, first calculate the ratio between the available uplink bandwidth of the first device and the video bitrate of the first device's video resources, and round down to obtain the maximum number of devices. Then, sort the direct data packet utilization rates of each second device in descending order, and determine the second utilization threshold as the utilization rate of the device with the largest number of devices in the sorted order. Alternatively, they can be sorted in ascending order, and the utilization rate of the device with the largest number of devices in the sorted order can be determined as the second utilization threshold.

[0145] In one possible implementation, please see Figure 3 The method further includes step 207. Wherein, Figure 3 Step 207 is shown to be executed after step 202. Optionally, step 207 may also be executed after step 203 or step 204, which is not limited here.

[0146] Step 207: Obtain the video resources of the first device and determine the video bitrate of the video resources of the first device; when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity and the second device does not meet the first transmission condition, send the video resources of the first device to the server so that the server forwards the video resources of the first device to the second device.

[0147] In this embodiment, if the first device determines that the utilization rate of the direct data packets from the second device to the first device is less than a second utilization threshold, the first device can send the identification information of the second device to the relay server. Furthermore, the first device sends its video resources to the relay server, enabling the relay server to forward the video resources of the first device to the second device based on the identification information of the second device. This allows the relay server to receive the video resources of the first device through the relay link even when the utilization rate of the direct data packets from the second device is low.

[0148] It should be noted that among other terminal devices participating in the same voice call as the first device, there may be terminal devices that have not established a direct connection link with the first device. For these terminal devices, the first device may send the identification information of these terminal devices to the relay server. After the first device sends the video resource of the first device to the relay server, the relay server forwards the video resource of the first device to these terminal devices based on the identification information of these terminal devices, which ensures that the terminal devices can receive the video resource of the first device even if no direct connection link is established with the first device.

[0149] In summary, for any terminal device participating in the same voice call as the first device, the terminal device may have established a direct connection link with the first device, and such terminal device may be the aforementioned second device, or the terminal device may not have established a direct connection link with the first device. For the first device i and the terminal device j, if a direct connection link is established between the first device i and the terminal device j, then C ij =1, and if no direct connection link is established between the first device i and the terminal device j, then C ij =0.

[0150] For video resources, a strategy of transmitting via direct connection links as much as possible is adopted to reduce the equipment cost and bandwidth cost of the relay server. Assume that there are Q terminal devices participating in the voice call, wherein the first device i acts as the main speaker, the available uplink bandwidth of the first device i is B, and the video bitrate of the video resource of the first device i is b. When any one of the following case B1 and case B2 occurs, it is necessary to use a relay link for transmitting the video resource.

[0151] Case B1:

[0152] In case B1, some of the Q terminal devices have not established a direct connection link with the first device i. Assume that no direct connection link is established between the first device i and the terminal device j, that is, C ij =0. At this time, the first device i sends the video resource of the first device i to the relay server, so that the relay server forwards the video resource of the first device i to the terminal device j.

[0153] Case B2: In the case where all Q-1 terminal devices have established direct connection links with the first device, B < b*(Q-1) is satisfied, which indicates that the available uplink bandwidth of the first device i cannot support simultaneously transmitting the video resource of the first device i to the other Q-1 terminal devices through the direct connection links. At this time, the first device i needs to use the broadcasting capability of the relay server to assist in completing the transmission of the video resource.

[0154] Assuming that after deducting the bandwidth required for device i to send its video resources to the relay link, device i's available uplink bandwidth can still support sending its video resources to T terminal devices via T direct links, then the utilization rate of direct data packets for the Q-1 terminal devices is... Sort the data in descending order and select the top T items. The corresponding terminal devices are designated as second devices that meet the first transmission condition. For these second devices that meet the first transmission condition, the first device sends its video resources to them via a direct link. The remaining QT-1 terminal devices are designated as second devices that do not meet the first transmission condition. For these terminal devices, the first device sends its video resources to a relay server, so that the relay server forwards the first device's time-frequency resources to these terminal devices.

[0155] By employing the above methods, the utilization rate of direct links is maximized while not exceeding the available uplink bandwidth of the first device, thereby improving the transmission quality of video resources. If situations B1 and B2 do not occur, the first device will not use relay links to transmit its video resources, effectively reducing server equipment and bandwidth costs.

[0156] It should be noted that the data from the first device can be divided into two categories. One category is critical data, characterized by high requirements for transmission quality (such as packet loss rate and transmission latency) and a relatively small data volume. During a voice call, audio resources and command data (such as encoding parameters when encoding audio and video resources) are considered critical data. The other category is non-critical data, characterized by lower requirements for transmission quality and a larger data volume. For example, during a voice call, video resources are considered non-critical data. In this embodiment, critical data employs a dual-transmission strategy using both relay links and direct links, as described above regarding the audio resources of the first device. Non-critical data, on the other hand, is transmitted primarily via direct links, as described above regarding the video resources of the first device. Of course, in practical applications, critical and non-critical data can also utilize other transmission methods, such as transmitting encoding parameters via signaling.

[0157] In one possible implementation, please see Figure 3 Step 202 may be followed by step 208. Optionally, step 208 may also be performed after any of steps 203 to 207.

[0158] Step 208: In response to a change in the target of the first device's role during the voice call, a direct connection closure request is sent to the second device via the direct connection link; via the direct connection link, a direct connection closure response is received from the second device in response to the direct connection closure request, and the direct connection link is closed based on the direct connection closure response.

[0159] In this embodiment, a direct link has been successfully established between the first device and the second device. Because this application employs an adaptive role switching strategy, the role of the first device may change when its device state changes. As the device state of the first device changes, its role may change accordingly. In this case, the first device needs to close the direct link with the second device; that is, the first device sends a request to close the direct link to the second device. At least two scenarios, C1 and C2, exist here.

[0160] In scenario C1, in response to a change in the target of the first device's role during a voice call, a request to close the direct connection is sent to the second device via the direct link. This includes: in response to a change in the device state corresponding to the first device to a state of disabled multimedia resource acquisition, determining that the target of the first device's role during the voice call has changed and that the first device's role during the voice call has changed to that of a listener. In this case, if the second device's role during the voice call is that of a listener, a request to close the direct connection is sent to the second device via the direct link.

[0161] When the first device disables video capture (e.g., disables screen sharing or camera) and / or audio capture (e.g., disables microphone), its device state changes from multimedia resource capture to multimedia resource capture disabled. At this point, the first device's role changes from speaker to listener; this role change represents a change in the target of the first device's role.

[0162] Since the first device acts as a listener, it is the receiver of multimedia resources and is primarily used to receive them. Because the second device, which also acts as a listener, does not collect multimedia resources, if the second device also acts as a listener, the first device will send a request to close the direct connection to the second device via the direct link, thus closing the direct link between them.

[0163] In scenario C2, in response to a change in the target of the first device's role during a voice call, a request to close the direct connection is sent to the second device via the direct connection link. This includes: in response to a change in the device state corresponding to the first device to an exited voice call state, determining that the target of the first device's role during the voice call has changed and that the first device's role during the voice call has changed to no role. In this case, a request to close the direct connection is sent to the second device via the direct connection link.

[0164] The first device can exit an ongoing voice call. At this point, the device's state changes from "participating in a voice call" to "exiting a voice call." The first device's role also changes from "listener," "speaker," or "intermediate" to "no role." This role change signifies a change in the first device's role objective.

[0165] When the first device's role is "no role," it is neither a recipient of multimedia resources nor a producer of multimedia information; that is, the first device neither receives nor collects multimedia resources. The first device can send a "close direct link" request to the second device via the direct link to close the direct link between them.

[0166] When the first device closes the direct link with the second device, it sends a request to the second device to close the direct link. Upon receiving the request, the second device sends a response to close the direct link with the first device and updates the link status of the direct link between them, changing it from open to closed. When the first device receives the response from the second device, it updates the link status of the direct link between them again, changing it from open to closed. At this point, the direct link between the first and second devices is closed.

[0167] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the audio resources and video resources involved in this application were obtained with full authorization.

[0168] In the above method, when the first device changes its role to speaker during a voice call, the second device is identified from at least one terminal device participating in the same voice call as the first device, and a direct link is created between the first and second devices, thus achieving dynamic creation of the direct link. On the one hand, the audio resources of the first device are sent to the second device through the direct link to reduce the transmission latency and packet loss rate of the audio resources. On the other hand, the audio resources of the first device are sent to the server so that the server can forward the audio resources of the first device to the second device, thereby improving the transmission quality of the audio resources.

[0169] by Figure 4 Taking the flowchart of another multimedia resource transmission method provided in this application embodiment as an example, the method can be... Figure 1 The method is executed by any terminal device (such as terminal device 2). Since the above embodiment involves at least two terminal devices, for ease of description, the terminal device executing the multimedia resource transmission method provided in this application embodiment is referred to as the first device, and the terminal devices other than the first device among the at least two terminal devices are referred to as other terminal devices. Figure 4 As shown, the method includes at least steps 401 to 403 as shown below.

[0170] Step 401: In response to receiving a direct connection request forwarded by the server from the first device, a direct connection link is created between the second device and the first device. The second device is determined from at least one terminal device participating in the same voice call as the first device when the first device changes its role to speaker during the voice call. The role of the first device during the voice call is related to the device state of the first device.

[0171] The details of step 401 can be found in the descriptions of steps 201 and 202 above. The implementation principles of the two are similar, and will not be repeated here.

[0172] In one possible implementation, creating a direct link between the second device and the first device includes: receiving a direct link request forwarded by a server for the second device, wherein the direct link request is sent by the first device to the server and carries the external network address of the first device; sending a direct link response to the direct link request to the server, wherein the direct link response carries the external network address of the second device; and creating a direct link between the second device and the first device based on the external network addresses of the first device and the second device.

[0173] The description of creating a direct link between the second device and the first device can be found in the description of "creating a direct link between the first device and the second device" above. The implementation principles of the two are similar, so they will not be repeated here.

[0174] Optionally, based on the external network address of the first device and the external network address of the second device, a direct link between the second device and the first device is created, including: sending multiple direct link data packets to the first device based on the external network address of the first device; and successfully creating a direct link between the second device and the first device in response to receiving a reception response from the first device based on the external network address of the second device for each direct link data packet.

[0175] For a description of successfully creating a direct link between the second device and the first device, please refer to the description of "successfully creating a direct link between the first device and the second device" above. The implementation principles of the two are similar, so they will not be repeated here.

[0176] In one possible implementation, after step 401, the method further includes: receiving a direct connection closure request sent by a first device via a direct connection link, wherein the direct connection closure request is sent by the first device to the second device when the target of the first device changes during a voice call; sending a direct connection closure response to the direct connection closure request to the first device via the direct connection link; and closing the direct connection link based on the direct connection closure response.

[0177] For a description of closing direct links, please refer to the description of "closing direct links" above. The implementation principles of the two are similar, so they will not be repeated here.

[0178] Step 402: Receive the audio resources of the first device sent by the first device through the direct link.

[0179] The description of step 402 can be found in the description of step 203 above. The implementation principles of the two are similar, and will not be repeated here.

[0180] Step 403: Receive the audio resources of the first device forwarded by the server.

[0181] The description of step 403 can be found in the description of step 204 above. The implementation principles of the two are similar, and will not be repeated here.

[0182] In one possible implementation, the method further includes step 404, receiving video resources of the first device sent by the first device via a direct link; the video resources of the first device are sent via the direct link when the available uplink bandwidth of the first device is not less than the product between the video bitrate of the video resources of the first device and the reference quantity.

[0183] The description of step 404 can be found in the description of step 205 above. The implementation principles of the two are similar, and will not be repeated here.

[0184] The method further includes step 405, receiving video resources of the first device sent by the first device through a direct link; the video resources of the first device are sent through the direct link when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity, and the direct data packet utilization rate of the second device is not less than the second utilization rate threshold, and the direct data packet utilization rate of the second device is used to characterize the proportion of data packets sent by the second device through the direct link.

[0185] The description of step 405 can be found in the description of step 206 above. The implementation principles of the two are similar, and will not be repeated here.

[0186] The method further includes step 406, receiving video resources of the first device forwarded by the server; the video resources of the first device are sent to the server when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity, and the utilization rate of the direct data packets of the second device is less than the second utilization threshold.

[0187] The description of step 406 can be found in the description of step 207 above. The implementation principles of the two are similar, and will not be repeated here.

[0188] In the above method, when the first device changes its role to speaker during a voice call, the second device is identified from at least one terminal device participating in the same voice call as the first device, and a direct link is created between the first and second devices, thus achieving dynamic creation of the direct link. On the one hand, the audio resources of the first device are sent to the second device through the direct link to reduce the transmission latency and packet loss rate of the audio resources. On the other hand, the audio resources of the first device are sent to the server so that the server can forward the audio resources of the first device to the second device, thereby improving the transmission quality of the audio resources.

[0189] This application also provides a multimedia resource transmission system, which includes a first device, a second device, and a server.

[0190] The first device is used to perform and Figure 2 , Figure 3 The method for transmitting related multimedia resources; the second device is used to perform the transmission of related multimedia resources. Figure 4 Related multimedia resource transmission methods; the server is used to perform and Figures 2 to 4 The functions performed by the relevant servers.

[0191] The above describes the method for transmitting multimedia resources provided in the embodiments of this application from the perspective of method steps. The following section, in conjunction with... Figures 5 to 7 To further describe. In the embodiments of this application, multiple terminal devices participate in the same voice call. Any one of the multiple terminal devices participating in the voice call is regarded as the first device, and the terminal devices other than the first device are regarded as other terminal devices.

[0192] The first device plays different roles during a voice call. These roles can be described using the term "roles of the first device." The role of the first device may change as its state changes. Please see [link / reference]. Figure 5 , Figure 5This is a schematic diagram of a role switching provided in an embodiment of this application.

[0193] When the first device is not participating in a voice call, its device state is "not participating in a voice call," and its role is "no role." When the first device participates in a voice call, its device state changes from "not participating in a voice call" to "participating in a voice call," and its role changes from "no role" to "listener." In listener mode, the first device can receive and play audio and / or receive and play video. Since the first device is not capturing audio and video at this time, its device state can also be described as "video resource capture disabled" and "audio resource capture disabled."

[0194] When the first device starts capturing video, its device status changes from "video resource capture off" to "video resource capture," and at this point, the first device's role changes from listener to speaker. Under the speaker's guidance, the first device can then record and send video.

[0195] When the first device begins acquiring audio, its device state changes from "audio resource acquisition off" to "audio resource acquisition." At this point, the first device's role shifts from listener to an intermediate state. In this intermediate state, the first device can receive and play video and / or receive and play audio. In this intermediate state, the first device can also record audio and perform voice activity detection, which lasts for a period of time (e.g., one minute). If no target audio is detected within this time, the first device reverts to listener status. If target audio is detected within this time, the first device changes to speaker status, and in this speaker role, the first device can record and send audio.

[0196] It should be noted that, while the device is in the speaker's position, it can also receive and play video and / or receive and play audio. When the first device starts capturing both audio and video simultaneously, its role changes directly from listener to speaker. If the first device stops capturing audio or video at this time, its role remains unchanged; that is, the first device continues to act as the speaker.

[0197] When the first device's device status is only in video resource acquisition mode, and the first device stops acquiring video, the device status changes from video resource acquisition mode to video resource acquisition disabled. When the first device's device status is only in audio resource acquisition mode, and the first device stops acquiring audio, the device status changes from audio resource acquisition mode to audio resource acquisition disabled. When the first device's device status is both video resource acquisition mode and audio resource acquisition mode, and the first device stops acquiring both audio and video, the device status changes from video resource acquisition mode to video resource acquisition disabled, and from audio resource acquisition mode to audio resource acquisition disabled. All three scenarios can be summarized as the first device stopping acquiring both audio and video; in this case, the first device's role changes from speaker to listener.

[0198] The first device can exit the voice call at any time during the call. When the first device exits the voice call, its device status changes from participating in the voice call to not participating in the voice call. At this time, the first device's role changes from listener, speaker, or intermediary to no role.

[0199] When the first device acts as a listener or an intermediary, a direct link can exist between the first device and other terminal devices whose role is that of the speaker, allowing the first device to receive and play audio and / or video. When the first device acts as the speaker, a direct link can exist between the first device and any other terminal device, allowing the first device to record and send audio and / or video, receive and play audio and / or video, and this terminal device may act as a listener, a speaker, or even an intermediary.

[0200] Because the first device has a different role, the other terminal devices that have direct links with the first device are also different. Therefore, the first device can create and close direct links in real time.

[0201] In this embodiment, when the role of the first device changes from no role to listener, the first device can establish a direct connection link with other terminal devices that have not established a direct connection link with the first device but whose role is the speaker. When the role of the first device changes from an intermediate state or listener to speaker, the first device can establish a direct connection link with other terminal devices that have not established a direct connection link with the first device. For an example of establishing a direct connection link between the first device and the second device, please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram illustrating the creation and closure of a direct link provided in an embodiment of this application.

[0202] When the first device participates in a voice call, it can create a direct connection socket and send UDP packets to the direct connection burrowing server based on this socket. The direct connection burrowing server parses the UDP packets to obtain the first device's external IP address and sends this address back to the first device, thus enabling the first device to obtain its external IP address. The first device can also obtain its external IP address in the same way.

[0203] When the first device needs to establish a direct link with the second device, the first device sends a direct link request to the relay server, which can forward the request to the second device. The direct link request includes the first device's external IP address. The second device parses the request to obtain and stores the first device's external IP address. Additionally, the second device sends a direct link response to the request to the relay server, which can forward this response to the first device. This response also includes the second device's external IP address. The first device parses the response to obtain and stores the second device's external IP address.

[0204] Next, a direct connection is established between the first and second devices. The first device sends multiple direct connection data packets to the second device based on the second device's external IP address. Each time the second device receives a direct connection data packet, it sends a receive response to the first device for that packet. When the first device receives the receive responses from the second device for each direct connection data packet, it determines that a direct connection link has been successfully established between the first and second devices.

[0205] At this time, the first device can send its multimedia resources to the second device via a direct link between the first and second devices, so that the second device can receive the multimedia resources from the first device. Similarly, the second device can also send its multimedia resources to the first device via a direct link between the first and second devices, so that the first device can receive the multimedia resources from the second device. Here, multimedia resources include audio resources and / or video resources.

[0206] When the first device needs to close the direct link with the second device, the first device sends a close direct link request to the second device through the direct link. Upon receiving the close direct link request, the second device sends a close direct link response to the first device through the direct link, confirming the closure of the direct link. Upon receiving the close direct link response, the first device confirms the closure of the direct link between itself and the second device.

[0207] Please see Figure 7 , Figure 7This is a schematic diagram illustrating the transmission of multimedia resources according to an embodiment of this application. In this embodiment, a first device can send its multimedia resources to a second device via a direct link between the first and second devices, so that the second device can receive the multimedia resources from the first device via the direct link. Alternatively, the first device can send its multimedia resources to a relay server, which then forwards the multimedia resources to the second device, allowing the second device to receive the multimedia resources from the first device via a relay link. Here, the multimedia resources are transmitted in the form of a stream; therefore, multimedia resources correspond to data packets.

[0208] Since the second device can receive multimedia resources from the first device via both a direct link and a relay link, it can receive each data packet twice. The second device uses the first received packet and discards the later received packets, thus deduplicating the received data packets. By analyzing the usage of statistical packets, the utilization rates of direct and relay packets can be obtained. The direct packet utilization rate is sent to the first device, and the relay packet utilization rate is sent to the relay server.

[0209] It should be noted that the multimedia resources of the first device include audio resources and video resources, and the embodiments of this application adopt different transmission strategies for audio resources and video resources.

[0210] For audio resources, a dual-transmission strategy using both relay links and direct links is employed. On one hand, the first device sends its audio resources to the second device via the direct link. On the other hand, the first device sends its audio resources to the relay server. The relay server determines whether the relay data packet utilization rate of the second device is not greater than a first utilization threshold. If the relay data packet utilization rate of the second device is not greater than the first utilization threshold, the relay server stops sending the first device's audio resources to the second device. If the relay data packet utilization rate of the second device is greater than the first utilization threshold, the relay server sends the first device's audio resources to the second device. This method ensures that the second device can receive the first device's audio resources while reducing invalid transmissions and conserving downlink resources.

[0211] For video resources, a strategy of transmitting through direct links as much as possible is adopted. Assume that there are Q terminal devices participating in the same voice call, the role of the first device m is the speaker, the available uplink bandwidth of m is B, the video bitrate of the video resource of m is b, the number of other terminal devices is Q-1, and direct links exist between the first device and each of the other terminal devices. Then, when B≥b*(Q-1), the first device sends its video resource to each of the other terminal devices through the direct link. When B<b*(Q-1), the first device sorts the utilization rates of direct link data packets for each of the other terminal devices in descending order, and determines the first T other terminal devices based on the sorting result. On one hand, the first device sends its video resource to the first T other terminal devices through the direct link, and on the other hand, the first device sends its video resource to the relay server, and the relay server sends the video resource of the first device to the remaining Q-T-1 terminal devices. Of course, if there is a terminal device that does not have a direct link established with the first device, the relay server will also send the video resource of the first device to this terminal device.

[0212] In this way, each of the other terminal devices participating in the voice call can receive the video resource of the first device through either a direct link or a relay link. In addition, by adopting the strategy of transmitting through direct links as much as possible, the equipment cost and bandwidth cost of the relay server can be reduced, and the transmission quality of video resources can be improved. While not exceeding the available uplink bandwidth of the first device, the direct links are fully utilized, and the overall transmission quality of the direct links is improved.

[0213] Figure 8 is a schematic structural diagram of a multimedia resource transmission apparatus provided in an embodiment of the present application, as shown in Figure 8 the figure, the apparatus includes:

[0214] a determining module 801, configured to, in response to the role of the first device being changed to the speaker during a voice call, determine a second device from at least one terminal device participating in the same voice call as the first device, where the role of the first device during the voice call is related to the device state of the first device;

[0215] a creating module 802, configured to create a direct link between the first device and the second device;

[0216] a sending module 803, configured to send the audio resource of the first device to the second device through the direct link;

[0217] the sending module 803, further configured to send the audio resource of the first device to a server, so that the server forwards the audio resource of the first device to the second device.

[0218] In a possible implementation, the apparatus further includes:

[0219] The acquisition module is used to acquire the audio resources acquired by the first device in response to the device state change of the first device to the audio resource acquisition state.

[0220] The determination module 801 is also used to determine the role of the first device as the speaker in response to detecting object audio in the audio resources collected by the first device.

[0221] In one possible implementation, module 802 is used to send a direct connection request for the second device to the server, the direct connection request carrying the external network address of the first device; receive a direct connection response forwarded by the server, the direct connection response being the response information sent by the second device to the server after receiving the direct connection request forwarded by the server, the direct connection response carrying the external network address of the second device; extract the external network address of the second device from the direct connection response; and create a direct connection link between the first device and the second device based on the external network addresses of the first device and the second device.

[0222] In one possible implementation, module 802 is created to send multiple direct connection data packets to the second device based on the external network address of the second device; in response to receiving a reception response from the second device based on the external network address of the first device for each direct connection data packet, a direct connection link between the first device and the second device is successfully created.

[0223] In one possible implementation, the server forwards the audio resources of the first device to the second device when a first forwarding condition is met. Meeting the first forwarding condition includes the second device's relay data packet utilization rate being greater than a first utilization rate threshold. The second device's relay data packet utilization rate is used to characterize the proportion of data packets used by the second device that are forwarded through the server. When the first forwarding condition is not met, the server stops forwarding the first device's audio resources to the second device.

[0224] In one possible implementation, the acquisition module is further configured to acquire the video resources of the first device and determine the video bitrate of the video resources of the first device.

[0225] The sending module 803 is further configured to send the video resources of the first device to the second device via a direct link when the available uplink bandwidth of the first device is not less than the product between the video bitrate of the video resources of the first device and the reference quantity.

[0226] In one possible implementation, the acquisition module is further configured to acquire the video resources of the first device and determine the video bitrate of the video resources of the first device.

[0227] The sending module 803 is further configured to send the video resources of the first device to the second device via a direct link when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity and the second device meets the first transmission condition.

[0228] The sending module 803 is further configured to send the video resources of the first device to the server when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity and the second device does not meet the first transmission condition, so that the server forwards the video resources of the first device to the second device.

[0229] In one possible implementation, the second device satisfies a first transmission condition, including: the direct data packet utilization rate of the second device is not less than a second utilization threshold, wherein the direct data packet utilization rate of the second device is used to characterize the proportion of data packets sent by the second device through the direct link.

[0230] In one possible implementation, the sending module 803 is further configured to send a request to close the direct connection to the second device via the direct connection link in response to a change in the target of the first device's role during the voice call.

[0231] The device also includes:

[0232] The receiving module is used to receive a close direct connection response from the second device in response to the close direct connection request via the direct connection link, and close the direct connection link based on the close direct connection response.

[0233] In one possible implementation, the sending module 803 is configured to, in response to a change in the device state corresponding to the first device to a state of "disabled multimedia resource acquisition", determine that the first device's role during the voice call has changed target and the role has changed to listener, and the second device's role during the voice call is also listener, then send a request to close the direct connection to the second device via the direct connection link; in response to a change in the device state corresponding to the first device to a state of "exited voice call", determine that the first device's role during the voice call has changed target and the role has changed to "no role", then send a request to close the direct connection to the second device via the direct connection link.

[0234] In the aforementioned device, when the first device changes its role to speaker during a voice call, a second device is identified from at least one terminal device participating in the same voice call as the first device, and a direct link is created between the first and second devices, thus achieving dynamic creation of the direct link. On one hand, the first device's audio resources are sent to the second device via the direct link to reduce audio resource transmission latency and packet loss rate. On the other hand, the first device's audio resources are sent to the server, enabling the server to forward the first device's audio resources to the second device, thereby improving the transmission quality of the audio resources.

[0235] It should be understood that the above Figure 8 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0236] Figure 9 This is a schematic diagram of the structure of a multimedia resource transmission device provided in an embodiment of this application, as shown below. Figure 9 As shown, the device includes:

[0237] Module 901 is used to respond to a direct connection request sent by the first device forwarded by the server and create a direct connection link between the second device and the first device. The second device is determined from at least one terminal device participating in the same voice call as the first device when the role of the first device changes to the speaker during the voice call. The role of the first device during the voice call is related to the device state of the first device.

[0238] The receiving module 902 is used to receive audio resources of the first device sent by the first device via a direct link;

[0239] The receiving module 902 is also used to receive audio resources from the first device forwarded by the server.

[0240] In one possible implementation, module 901 is created to receive a direct connection request forwarded by the server for the second device, wherein the direct connection request is sent by the first device to the server and carries the external network address of the first device; to send a direct connection response to the direct connection request to the server, wherein the direct connection response carries the external network address of the second device; and to create a direct connection link between the second device and the first device based on the external network addresses of the first device and the second device.

[0241] In one possible implementation, module 901 is created to send multiple direct connection data packets to the first device based on the external network address of the first device; in response to receiving a reception response from the first device based on the external network address of the second device for each direct connection data packet, a direct connection link between the second device and the first device is successfully created.

[0242] In one possible implementation, the receiving module 902 is further configured to receive video resources of the first device sent by the first device via a direct link; the video resources of the first device are sent via the direct link when the available uplink bandwidth of the first device is not less than the product between the video bitrate of the video resources of the first device and the reference quantity.

[0243] In one possible implementation, the receiving module 902 is further configured to receive video resources of the first device sent by the first device via a direct link; the video resources of the first device are sent via the direct link when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity, and the direct data packet utilization rate of the second device is not less than a second utilization threshold; the direct data packet utilization rate of the second device is used to characterize the proportion of data packets sent by the second device via the direct link.

[0244] In one possible implementation, the receiving module 902 is further configured to receive video resources of the first device forwarded by the server; the video resources of the first device are sent to the server when the available uplink bandwidth of the first device is less than the product between the video bitrate of the video resources of the first device and the reference quantity, and the utilization rate of the direct data packets of the second device is less than a second utilization threshold.

[0245] In one possible implementation, the receiving module 902 is further configured to receive a direct connection closure request sent by the first device via a direct connection link. The direct connection closure request is sent by the first device to the second device when the target of the first device changes during a voice call.

[0246] The device also includes:

[0247] The sending module is used to send a close direct connection response to the close direct connection request to the first device via the direct connection link, and close the direct connection link based on the close direct connection response.

[0248] In the aforementioned device, when the first device changes its role to speaker during a voice call, a second device is identified from at least one terminal device participating in the same voice call as the first device, and a direct link is created between the first and second devices, thus achieving dynamic creation of the direct link. On one hand, the first device's audio resources are sent to the second device via the direct link to reduce audio resource transmission latency and packet loss rate. On the other hand, the first device's audio resources are sent to the server, enabling the server to forward the first device's audio resources to the second device, thereby improving the transmission quality of the audio resources.

[0249] It should be understood that the above Figure 9The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0250] Figure 10 A structural block diagram of a terminal device 1300 provided in an exemplary embodiment of this application is shown. The terminal device 1300 includes a processor 1301 and a memory 1302.

[0251] Processor 1301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0252] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1302 are used to store at least one computer program, which is executed by the processor 1301 to implement the multimedia resource transmission method provided in the method embodiments of this application.

[0253] In some embodiments, the terminal device 1300 may also optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, memory 1302, and peripheral device interface 1303 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, and a power supply 1308.

[0254] Peripheral device interface 1303 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1301 and memory 1302. In some embodiments, processor 1301, memory 1302 and peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1301, memory 1302 and peripheral device interface 1303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0255] The radio frequency (RF) circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1304 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1304 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0256] Display screen 1305 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1305 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1301 for processing. In this case, display screen 1305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 1305 may be a single screen, disposed on the front panel of terminal device 1300; in other embodiments, display screen 1305 may be at least two, disposed on different surfaces of terminal device 1300 or in a folded design; in still other embodiments, display screen 1305 may be a flexible display screen, disposed on a curved or folded surface of terminal device 1300. Furthermore, display screen 1305 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0257] The camera assembly 1306 is used to acquire images or videos. Optionally, the camera assembly 1306 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0258] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1301 for processing, or input to the radio frequency circuit 1304 to achieve voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal device 1300. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1307 may also include a headphone jack.

[0259] Power supply 1308 is used to supply power to the various components in terminal device 1300. Power supply 1308 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1308 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0260] In some embodiments, the terminal device 1300 further includes one or more sensors 1309. The one or more sensors 1309 include, but are not limited to: an acceleration sensor 1311, a gyroscope sensor 1312, a pressure sensor 1313, an optical sensor 1314, and a proximity sensor 1315.

[0261] Accelerometer 1311 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 1300. For example, accelerometer 1311 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1301 can control display screen 1305 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1311. Accelerometer 1311 can also be used for games or for acquiring user motion data.

[0262] The gyroscope sensor 1312 can detect the orientation and rotation angle of the terminal device 1300. The gyroscope sensor 1312 can work in conjunction with the accelerometer sensor 1311 to collect 3D motion data from the user on the terminal device 1300. Based on the data collected by the gyroscope sensor 1312, the processor 1301 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0263] The pressure sensor 1313 can be disposed on the side bezel of the terminal device 1300 and / or on the lower layer of the display screen 1305. When the pressure sensor 1313 is disposed on the side bezel of the terminal device 1300, it can detect the user's grip signal on the terminal device 1300, and the processor 1301 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1313. When the pressure sensor 1313 is disposed on the lower layer of the display screen 1305, the processor 1301 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1305. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0264] Optical sensor 1314 is used to collect ambient light intensity. In one embodiment, processor 1301 can control the display brightness of display screen 1305 based on the ambient light intensity collected by optical sensor 1314. Specifically, when the ambient light intensity is high, the display brightness of display screen 1305 is increased; when the ambient light intensity is low, the display brightness of display screen 1305 is decreased. In another embodiment, processor 1301 can also dynamically adjust the shooting parameters of camera assembly 1306 based on the ambient light intensity collected by optical sensor 1314.

[0265] The proximity sensor 1315, also known as a distance sensor, is typically located on the front panel of the terminal device 1300. The proximity sensor 1315 is used to detect the distance between the user and the front of the terminal device 1300. In one embodiment, when the proximity sensor 1315 detects that the distance between the user and the front of the terminal device 1300 is gradually decreasing, the processor 1301 controls the display screen 1305 to switch from a screen-on state to a screen-off state; when the proximity sensor 1315 detects that the distance between the user and the front of the terminal device 1300 is gradually increasing, the processor 1301 controls the display screen 1305 to switch from a screen-off state to a screen-on state.

[0266] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the terminal device 1300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0267] Figure 11This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more processors 1401 and one or more memories 1402. The one or more memories 1402 store at least one computer program, which is loaded and executed by the one or more processors 1401 to implement the multimedia resource transmission method provided in the above-described method embodiments. For example, the processor 1401 is a CPU. Of course, the server 1400 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1400 may also include other components for implementing device functions, which will not be elaborated here.

[0268] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for transmitting multimedia resources.

[0269] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0270] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for transmitting multimedia resources.

[0271] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0272] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0273] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for transmitting multimedia resources, characterized in that, The method includes: In response to the first device changing its role to speaker during a voice call, a second device is determined among at least one terminal device participating in the same voice call as the first device, wherein the role of the first device during the voice call is related to the device state of the first device; Create a direct link between the first device and the second device; The audio resources of the first device are sent to the second device via the direct link; Send the audio resources of the first device to the server so that the server forwards the audio resources of the first device to the second device. Stop forwarding the audio resources of the first device to the second device when the utilization rate of the relay data packets of the second device is not greater than a first utilization threshold. The relay data packet utilization rate of the second device is obtained by counting the number of data packets forwarded by the server and the number of data packets received by the second device through the direct link. Any data packet used by the second device is the audio resource received first among the audio resources forwarded by the server and the same audio resources received through the direct link.

2. The method according to claim 1, characterized in that, Before determining the second device among at least one terminal device participating in the same voice call as the first device in response to the first device changing its role to speaker during a voice call, the method further includes: In response to the device state of the first device changing to audio resource acquisition state, the audio resources acquired by the first device are obtained; In response to detecting object audio in the audio resources collected by the first device, it is determined that the role of the first device changes to speaker during the voice call.

3. The method according to claim 1, characterized in that, Creating a direct link between the first device and the second device includes: Send a direct connection request for the second device to the server, wherein the direct connection request carries the external network address of the first device; The device receives a direct connection response forwarded by the server. The direct connection response is a response message sent by the second device to the server in response to the direct connection request after receiving the direct connection request forwarded by the server. The direct connection response carries the external network address of the second device. Extract the external network address of the second device from the direct connection response; A direct link is created between the first device and the second device based on the external network address of the first device and the external network address of the second device.

4. The method according to claim 3, characterized in that, The step of creating a direct link between the first device and the second device based on the external network address of the first device and the external network address of the second device includes: Based on the external network address of the second device, send multiple direct connection data packets to the second device; Upon receiving a response from the second device based on the external network address of the first device for each directly connected data packet, a direct connection link between the first device and the second device is successfully established.

5. The method according to claim 1, characterized in that, The server forwards the audio resources of the first device to the second device when the utilization rate of the relay data packets of the second device is greater than the first utilization threshold.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Acquire the video resources of the first device and determine the video bitrate of the video resources of the first device; When the available uplink bandwidth of the first device is not less than the product of the video bitrate of the video resources of the first device and the reference quantity, the video resources of the first device are sent to the second device through the direct link.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Acquire the video resources of the first device and determine the video bitrate of the video resources of the first device; When the available uplink bandwidth of the first device is less than the product of the video bitrate and the reference quantity of the video resources of the first device, and the second device meets the first transmission condition, the video resources of the first device are sent to the second device through the direct link. When the available uplink bandwidth of the first device is less than the product of the video bitrate of the video resource of the first device and the reference quantity, and the second device does not meet the first transmission condition, the video resource of the first device is sent to the server so that the server forwards the video resource of the first device to the second device.

8. The method according to claim 7, characterized in that, The second device satisfies the first transmission condition, including: The direct data packet utilization rate of the second device is not less than the second utilization threshold. The direct data packet utilization rate of the second device is used to characterize the proportion of data packets sent by the second device through the direct link.

9. The method according to any one of claims 1 to 5, characterized in that, After establishing the direct link between the first device and the second device, the method further includes: In response to a change in the target of the first device's role during a voice call, a request to close the direct connection is sent to the second device via the direct connection link; Through the direct link, the system receives a direct link closure response from the second device in response to the direct link closure request, and closes the direct link based on the direct link closure response.

10. The method according to claim 9, characterized in that, The step of responding to a change in the target role of the first device during a voice call by sending a request to close the direct connection to the second device via the direct connection link includes: In response to the device state change of the first device to the state of disabling multimedia resource acquisition, it is determined that the role of the first device in the voice call process has changed and the role has changed to listener. If the role of the second device in the voice call process is listener, a request to disable direct connection is sent to the second device through the direct connection link. In response to the device status change of the first device to exit the voice call state, and determining that the role of the first device during the voice call has changed and the role has changed to no role, a request to close the direct connection is sent to the second device through the direct connection link.

11. A method for transmitting multimedia resources, characterized in that, The method includes: In response to receiving a direct connection request sent by the first device forwarded by the server, a direct connection link is created between the second device and the first device. The second device is determined from at least one terminal device participating in the same voice call as the first device when the role of the first device changes to speaker during the voice call. The role of the first device during the voice call is related to the device state of the first device. The audio resources of the first device are received via the direct link; Receive the audio resources of the first device forwarded by the server; From the audio resources forwarded by the server and the same audio resources received through the direct link, select the audio resource received first to obtain any data packet used by the second device; The relay data packet utilization rate of the second device is obtained by counting the number of data packets forwarded by the server and the number of data packets received by the second device through the direct link. The server sends the relay data packet utilization rate of the second device to the server so that the server stops forwarding the audio resources of the first device to the second device when the relay data packet utilization rate of the second device is not greater than a first utilization threshold.

12. The method according to claim 11, characterized in that, The method further includes: The video resources of the first device sent by the first device are received through the direct link; The video resources of the first device are transmitted via the direct link when the available uplink bandwidth of the first device is not less than the product of the video bitrate of the video resources of the first device and the reference quantity.

13. The method according to claim 11, characterized in that, The method further includes: The video resources of the first device sent by the first device are received through the direct link; The video resources of the first device are sent through the direct link when the available uplink bandwidth of the first device is less than the product of the video bitrate and the reference quantity of the video resources of the first device, and the direct data packet utilization of the second device is not less than the second utilization threshold. The direct data packet utilization of the second device is used to characterize the proportion of data packets sent through the direct link by the second device.

14. The method according to claim 11, characterized in that, The method further includes: Receive video resources from the first device forwarded by the server; The video resources of the first device are sent to the server when the available uplink bandwidth of the first device is less than the product of the video bitrate and the reference quantity of the video resources of the first device, and the direct data packet utilization of the second device is less than the second utilization threshold.

15. The method according to claim 11, characterized in that, After establishing the direct link between the second device and the first device, the method further includes: Through the direct connection link, a direct connection shutdown request sent by the first device is received. The direct connection shutdown request is sent by the first device to the second device when the target of the first device changes during a voice call. Through the direct link, a close direct link response is sent to the first device in response to the close direct link request, and the direct link is closed based on the close direct link response.

16. A multimedia resource transmission device, characterized in that, The device includes: The determination module is used to determine a second device among at least one terminal device participating in the same voice call as the first device in response to the first device changing its role to speaker during the voice call. The role of the first device during the voice call is related to the device state of the first device. A creation module is used to create a direct link between the first device and the second device; The sending module is used to send the audio resources of the first device to the second device via the direct connection link; The sending module is further configured to send the audio resources of the first device to the server, so that the server forwards the audio resources of the first device to the second device, and stops forwarding the audio resources of the first device to the second device when the utilization rate of the relay data packets of the second device is not greater than a first utilization rate threshold. The relay data packet utilization rate of the second device is obtained by counting the number of data packets forwarded by the server and the number of data packets received by the second device through the direct link. Any data packet used by the second device is the audio resource received first among the audio resources forwarded by the server and the same audio resources received through the direct link.

17. The apparatus according to claim 16, characterized in that, The device further includes: The acquisition module is used to acquire the audio resources acquired by the first device in response to the device state of the first device changing to the audio resource acquisition state. The determining module is further configured to determine, in response to detecting object audio in the audio resources collected by the first device, that the role of the first device has changed to speaker during the voice call.

18. The apparatus according to claim 16, characterized in that, The creation module is configured to send a direct connection request for the second device to the server, the direct connection request carrying the external network address of the first device; receive a direct connection response forwarded by the server, the direct connection response being the response information sent by the second device to the server after receiving the direct connection request forwarded by the server, the direct connection response carrying the external network address of the second device; extract the external network address of the second device from the direct connection response; and create a direct connection link between the first device and the second device based on the external network addresses of the first device and the second device.

19. The apparatus according to claim 18, characterized in that, The creation module is used to send multiple direct connection data packets to the second device based on the external network address of the second device; in response to receiving a reception response from the second device based on the external network address of the first device for each direct connection data packet, a direct connection link between the first device and the second device is successfully created.

20. The apparatus according to claim 16, characterized in that, The server forwards the audio resources of the first device to the second device when the utilization rate of the relay data packets of the second device is greater than the first utilization threshold.

21. The apparatus according to any one of claims 16 to 20, characterized in that, The device further includes: The acquisition module is used to acquire the video resources of the first device and determine the video bitrate of the video resources of the first device. The sending module is further configured to send the video resources of the first device to the second device via the direct link when the available uplink bandwidth of the first device is not less than the product between the video bitrate of the video resources of the first device and the reference quantity.

22. The apparatus according to any one of claims 16 to 20, characterized in that, The device further includes: The acquisition module is used to acquire the video resources of the first device and determine the video bitrate of the video resources of the first device. The sending module is further configured to send the video resources of the first device to the second device through the direct link when the available uplink bandwidth of the first device is less than the product between the video bitrate and the reference quantity of the video resources of the first device and the second device meets the first transmission condition. The sending module is further configured to send the video resources of the first device to the server when the available uplink bandwidth of the first device is less than the product of the video bitrate of the video resources of the first device and the reference quantity, and the second device does not meet the first transmission condition, so that the server forwards the video resources of the first device to the second device.

23. The apparatus according to claim 22, characterized in that, The second device satisfies the first transmission condition, including: The direct data packet utilization rate of the second device is not less than the second utilization threshold. The direct data packet utilization rate of the second device is used to characterize the proportion of data packets sent by the second device through the direct link.

24. The apparatus according to any one of claims 16 to 20, characterized in that, The sending module is also configured to send a request to close the direct connection to the second device via the direct connection link in response to a change in the target of the first device's role during a voice call. The device further includes: The receiving module is configured to receive, via the direct link, a close direct link response sent by the second device in response to the close direct link request, and close the direct link based on the close direct link response.

25. The apparatus according to claim 24, characterized in that, The sending module is used to respond to the device state change of the first device to the state of disabling multimedia resource acquisition, determine that the role of the first device in the voice call process has changed and the role has changed to listener, and if the role of the second device in the voice call process is listener, then send a request to disable direct connection to the second device through the direct connection link. In response to the device status change of the first device to exit the voice call state, and determining that the role of the first device during the voice call has changed and the role has changed to no role, a request to close the direct connection is sent to the second device through the direct connection link.

26. A multimedia resource transmission device, characterized in that, The device includes: A creation module is used to respond to a direct connection request sent by a first device forwarded by a server and create a direct connection link between a second device and the first device. The second device is determined from at least one terminal device participating in the same voice call as the first device when the role of the first device changes to speaker during a voice call. The role of the first device during the voice call is related to the device state of the first device. The receiving module is used to receive audio resources of the first device sent by the first device through the direct connection link; The receiving module is also used to receive the audio resources of the first device forwarded by the server; The sending module is configured to select the first received audio resource from the audio resources forwarded by the server and the same audio resources received through the direct link, to obtain any data packet used by the second device; count the number of data packets forwarded by the server and the number of data packets received through the direct link by the second device to obtain the relay data packet utilization rate of the second device; and send the relay data packet utilization rate of the second device to the server so that the server stops forwarding the audio resources of the first device to the second device when the relay data packet utilization rate of the second device is not greater than a first utilization threshold.

27. The apparatus according to claim 26, characterized in that, The receiving module is further configured to receive video resources of the first device sent by the first device via the direct link; The video resources of the first device are transmitted via the direct link when the available uplink bandwidth of the first device is not less than the product of the video bitrate of the video resources of the first device and the reference quantity.

28. The apparatus according to claim 26, characterized in that, The receiving module is further configured to receive video resources of the first device sent by the first device via the direct link; The video resources of the first device are sent through the direct link when the available uplink bandwidth of the first device is less than the product of the video bitrate and the reference quantity of the video resources of the first device, and the direct data packet utilization of the second device is not less than the second utilization threshold. The direct data packet utilization of the second device is used to characterize the proportion of data packets sent through the direct link by the second device.

29. The apparatus according to claim 26, characterized in that, The receiving module is also used to receive video resources from the first device forwarded by the server; The video resources of the first device are sent to the server when the available uplink bandwidth of the first device is less than the product of the video bitrate and the reference quantity of the video resources of the first device, and the direct data packet utilization of the second device is less than the second utilization threshold.

30. The apparatus according to claim 26, characterized in that, The receiving module is further configured to receive a direct connection closure request sent by the first device via the direct connection link, wherein the direct connection closure request is sent by the first device to the second device when the target of the role changes during the voice call. The device further includes: The sending module is configured to send a close direct connection response to the close direct connection request to the first device via the direct connection link, and close the direct connection link based on the close direct connection response.

31. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the method for transmitting multimedia resources as described in any one of claims 1 to 15.

32. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement the method for transmitting multimedia resources as described in any one of claims 1 to 15.

33. A multimedia resource transmission system, characterized in that, The system includes a first device, a second device, and a server; The first device is used to perform the multimedia resource transmission method as described in any one of claims 1 to 10; The second device is used to perform the method for transmitting multimedia resources as described in any one of claims 11 to 15; The server is used to perform the functions performed by the server as described in any one of claims 1 to 15.

34. A computer program product, characterized in that, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the method for transmitting multimedia resources as described in any one of claims 1 to 15.

Images