Conference participation method, apparatus, terminal device, and computer program product

Abstract

The application provides a conference participation method, device, terminal equipment and computer program product. The method comprises: a first terminal equipment acquires a first instruction, and then sends a second instruction to a second terminal equipment according to the first instruction. Wherein a first user enters a first conference according to a conference application in the first terminal equipment, the conference application comprises a plurality of modules with different functions, the first instruction instructs to replace a first module in the plurality of modules with the second terminal equipment to process first data, the first data is part or all conference data related to the function of the first module in the first conference; the second instruction instructs the second terminal equipment to run a second module in the second terminal equipment to process the first data, the second module and the first module are the same in function, and the first data processed by the second terminal equipment is sent by the first terminal equipment / conference server to the second terminal equipment or collected by the second terminal equipment. The above scheme of the application can improve the user participation experience.

Description

Technical Field

[0001] This application relates to the field of distributed technology, and in particular to a meeting participation method, apparatus, terminal equipment, and computer program product. Background Technology

[0002] With the rapid development of internet technology, online meetings have become a common communication method, widely used in scenarios such as daily office work and online education.

[0003] Currently, users need to participate in online meetings through a conferencing application (APP). Specifically, users first install and run the conferencing APP on one of their terminal devices, and then enter the meeting ID, meeting password, username, and other information on the APP's user interface to join the online meeting corresponding to that meeting ID.

[0004] During the meeting, if a user has other suitable devices nearby (e.g., larger screens or better speaker quality), the user needs to exit the meeting on their current device, install and run the same meeting app on the other device, and then rejoin the meeting with the same meeting ID through the other device's meeting app. This process of exiting and rejoining is cumbersome and may cause users to miss some meeting content, resulting in a poor meeting experience. Summary of the Invention

[0005] This application provides a meeting participation method, apparatus, terminal equipment, and computer program product that can enhance the user's meeting experience.

[0006] Firstly, this application provides a meeting participation method applied to a first terminal device. The method includes: the first terminal device acquiring a first instruction, and then the first terminal device sending a second instruction to a second terminal device according to the first instruction. Specifically, a first user enters a first meeting via a meeting application on the first terminal device. The meeting application includes multiple modules with different functions. The first instruction instructs the second terminal device to replace the first module among the multiple modules to process first data. The first data is part or all of the meeting data in the first meeting related to the function of the first module. The second instruction instructs the second terminal device to run a second module in the second terminal device to process the first data. The second module has the same function as the first module. The first data processed by the second terminal device is either sent to the second terminal device by the first terminal device / meeting server or collected by the second terminal device.

[0007] In the above solution, the first terminal device is the device used by the first user to join the meeting, and using only the first terminal device may result in a poor meeting experience. To improve the meeting experience, the second module (with the same function as the first module) in the second terminal device can replace the first module in the first terminal device to process some or all of the meeting data related to the function of the first module. In other words, some or all of the functions of the first module of the meeting application in the first terminal device are transferred to the second terminal device. At this time, the first user participates in the meeting using both the first and second terminal devices, which can give full play to the advantages of different terminal devices and thus improve the user's meeting experience. Furthermore, since some or all of the meeting data related to the function of the first module is processed on the second terminal device, the processor load of the first terminal device is reduced, thereby reducing the power consumption and heat generation of the first terminal device.

[0008] Based on the first aspect, in a possible implementation, in response to a first operation by a first user on a first terminal device, the first terminal device generates a first instruction, wherein the first operation includes instructing an operation of a first module and instructing an operation of a second terminal device. Alternatively, the first terminal device receives a first instruction from a second terminal device, wherein the first instruction is generated by the second terminal device based on a second operation by the first user on the second terminal device, and the second operation includes instructing an operation of the first module.

[0009] In the above scheme, the first user can directly operate on the first terminal device to instruct the first terminal device to use the second terminal device to replace the first module in processing some or all of the conference data related to the function of the first module. Alternatively, the first user can operate on the second terminal device, which generates a first instruction and sends it to the first terminal device, thereby indirectly instructing the first terminal device to use the second terminal device to replace the first module in processing some or all of the data related to the function of the first module.

[0010] Based on the first aspect, in possible implementations, the operation of instructing the first module includes one or more of the following: selecting the hardware required for the first module to run, selecting the identifier of the first module, and performing the somatosensory operation corresponding to the first module.

[0011] Based on the first aspect, in possible implementations, the aforementioned hardware includes one or more of a microphone, speaker, camera, screen, processor, and sensor.

[0012] Based on the first aspect, in a possible implementation, after the first terminal device obtains the first instruction, the method further includes: if the first data is all conference data related to the function of the first module in the first conference, the first terminal device obtains the Internet Protocol (IP) address of the second terminal device and the port corresponding to the second module in the second terminal device; then, the first terminal device sends parameter update information to the conference server; subsequently, the first terminal device receives confirmation information sent by the conference server. The parameter update information includes the IP address of the second terminal device and the port corresponding to the second module in the second terminal device; the confirmation information instructs the conference server to change the recorded first correspondence to a second correspondence. The first correspondence is the correspondence between the IP addresses of the first user and the first terminal device and the port corresponding to the first module in the first terminal device; the second correspondence is the correspondence between the IP addresses of the first user and the second terminal device and the port corresponding to the second module in the second terminal device.

[0013] In the above scheme, the first terminal device is the first user's access device. At the beginning of the access, the first terminal device and the conference server negotiate some parameters. These parameters are used to indicate the data interaction method between the first user and the conference server. The first correspondence is part of the information recorded by the conference server during the parameter negotiation process. The first correspondence indicates that the first user and the conference server transmit conference data related to the function of the first module through the IP address of the first terminal device and the port corresponding to the first module.

[0014] When the first terminal device receives the first instruction, which instructs the second terminal device to replace the first module in processing all conference data related to the function of the first module (i.e., to transfer all functions of the first module to the second terminal device), there is no longer a need for the first terminal device and the conference server to transmit conference data related to the function of the first module. Instead, the data can be transmitted between the conference server and the second terminal device. This change can be achieved through the following steps: The first terminal device sends parameter update information to the conference server, instructing the conference server to modify the recorded first correspondence to a second correspondence. The second correspondence indicates that the first user and the conference server transmit conference data related to the function of the first module through the IP address of the second terminal device and the port corresponding to the second module. When the first terminal device receives confirmation information, it indicates that the conference server has completed the modification. Subsequently, the conference server and the first user will transmit conference data related to the function of the first module through the IP address of the second terminal device and the port corresponding to the second module. The conference server and the first terminal device will no longer transmit conference data related to the function of the first module.

[0015] Based on the first aspect, in a possible implementation, after the first terminal device receives the confirmation information sent by the conference server, the first terminal device may stop running the first module to reduce the power consumption and heat generation of the first terminal device.

[0016] Based on the first aspect, in a possible implementation, after the first terminal device receives the confirmation information sent by the conference server, the first terminal device may send parameters related to the first module to the second terminal device, wherein the parameters related to the first module include one or more of the following: encoding / decoding protocol type, resolution, bandwidth, transmission protocol type, IP address of the conference server, and port.

[0017] It should be understood that when the first user joins the meeting through the first terminal device, the first terminal device and the conference server negotiate some parameters. These parameters are used to indicate the data transmission method between the first user and the conference server, including the parameters related to the first module mentioned above. When it is confirmed that the second module in the second terminal device will replace the first module to achieve the corresponding function, the first terminal device can send the previously recorded parameters related to the first module to the second terminal device. This allows the second terminal device to interact with the conference server based on the parameters related to the first module, eliminating the need for the first terminal device to forward data between the second terminal device and the conference server, thus helping to reduce the processor load and power consumption of the first terminal device.

[0018] Based on the first aspect, in a possible implementation, after the first terminal device sends a second instruction to the second terminal device according to the first instruction, the method further includes: if the first data processed by the second terminal device is collected by the second terminal device, the first terminal device receives the processed first data sent by the second terminal device and sends the processed first data to the conference server; if the first data processed by the second terminal device is sent by the first terminal device, the first terminal device receives the first data sent by the conference server and sends the first data to the second terminal device.

[0019] In the above scheme, when it is determined that a second terminal device will replace the first module in processing some or all of the conference data related to the functions of the first module, the second terminal device can send the processed conference data to the first terminal device, which will then forward it to the conference server; or, the first terminal device can receive the conference data related to the functions of the first module sent by the conference server, and then send the conference data to the second terminal device, which will then process the conference data using the second module. In other words, the first terminal device is responsible for forwarding data between the conference server and the second terminal device, but the interaction between the conference server and the first terminal device remains unchanged. The conference server will not be aware that the second terminal device is replacing the first module in processing some or all of the conference data related to the functions of the first module, and there is no need to renegotiate the parameters used to indicate the data transmission method between the conference server and the first user, thus avoiding the hassle of renegotiation.

[0020] Based on the first aspect, in a possible implementation, after the first terminal device receives the first instruction, it can copy the first module to the second terminal device, thus obtaining a second module in the second terminal device. In other words, the first terminal device can copy the first module to install a module with the same function as the first module on the second terminal device, enabling the second terminal device to process meeting data related to the function of the first module. This eliminates the need for the second terminal device to download modules related to the function of the first module from app stores or official application websites, making it more convenient and faster.

[0021] Based on the first aspect, in a possible implementation, before the first terminal device obtains the first instruction, the first terminal device receives device information sent by the second terminal device. The device information indicates that the second terminal device supports running the first module. That is, the second terminal device can obtain its own device information in advance to determine whether it can run the first module. If so, a module with the same function as the first module (i.e., the second module) can be installed on the second terminal device, and the second terminal device can run the second module, thereby using the second module in the second terminal device to replace the first module in processing meeting data related to the function of the first module. If not, the second terminal device cannot replace the first module in processing meeting data related to the function of the first module.

[0022] Based on the first aspect, in possible implementations, the first module includes one or more of the following: a conference control module, a conference collaboration module, an artificial intelligence processing module, an audio acquisition and encoding module, an audio decoding and playback module, a video acquisition and encoding module, and a video decoding and rendering module.

[0023] Secondly, this application provides another method for participating in a meeting, applied to a second terminal device. The method includes: the second terminal device receiving a second instruction sent by a first terminal device according to a first instruction; and then the second terminal device running a second module in the second terminal device according to the second instruction to process first data. Specifically, the first user enters the first meeting through a meeting application in the first terminal device. The meeting application includes multiple modules with different functions. The first instruction instructs the second terminal device to replace the first module among the multiple modules to process the first data. The first data is part or all of the meeting data in the first meeting related to the function of the first module. The second module has the same function as the first module, and the first data processed by the second terminal device is either sent to the second terminal device by the first terminal device / meeting server or collected by the second terminal device.

[0024] Based on the second aspect, in a possible implementation, before the second terminal device receives the second instruction sent by the first terminal device according to the first instruction, the method further includes: in response to a second operation by the first user on the second terminal device, the second terminal device generates the first instruction, the second operation including instructing the operation of the first module, and then the second terminal device sends the first instruction to the first terminal device.

[0025] Based on the second aspect, in possible implementations, the operation of instructing the first module includes one or more of the following: selecting the hardware required for the first module to run, selecting the identifier of the first module, and performing the somatosensory operation corresponding to the first module.

[0026] Based on the second aspect, in possible implementations, the aforementioned hardware includes one or more of a microphone, speaker, camera, screen, processor, and sensor.

[0027] Based on the second aspect, in a possible implementation, when the first data is all conference data related to the function of the first module in the first conference, the second terminal device receives parameters related to the first module sent by the first terminal device, and then the second terminal device processes the first data according to the parameters related to the first module and the second module. The parameters related to the first module include one or more of the following: encoding / decoding protocol type, resolution, bandwidth, transmission protocol type, IP address of the conference server, and port.

[0028] Based on the second aspect, in possible implementations, when the first data is part of the meeting data related to the function of the first module in the first meeting and the first data processed by the second terminal device is collected by the second terminal device, the second terminal device sends the first data processed according to the second module to the first terminal device; when the first data is part of the meeting data related to the function of the first module in the first meeting and the first data processed by the second terminal device is sent by the first terminal device, the second terminal device receives the first data sent by the first terminal device.

[0029] Based on the second aspect, in a possible implementation, the second module is obtained by the first terminal device copying the first module onto the second terminal device.

[0030] Based on the second aspect, in a possible implementation, before the second terminal device receives the second instruction sent by the first terminal device according to the first instruction, the second terminal device may send device information to the first terminal device, the device information indicating that the second terminal device supports running a module with the same function as the first module.

[0031] Based on the second aspect, in possible implementations, the first module includes one or more of the following: a conference control module, a conference collaboration module, an artificial intelligence processing module, an audio acquisition and encoding module, an audio decoding and playback module, a video acquisition and encoding module, and a video decoding and rendering module.

[0032] Thirdly, this application also provides a meeting participation device, including a unit for performing a method as described in any of the possible implementations of the first or second aspect.

[0033] Fourthly, this application also provides a terminal device, including a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the terminal device to perform a method as described in any possible implementation of the first or second aspect.

[0034] Fifthly, this application also provides a computer-readable storage medium including computer program instructions that, when executed by a terminal device, cause the terminal device to perform a method as described in any of the possible embodiments of the first or second aspect.

[0035] Sixthly, this application also provides a computer program product containing instructions. When the aforementioned instructions are executed by a terminal device, the terminal device performs a method as described in any of the possible implementations of the first or second aspect.

[0036] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments are briefly introduced below.

[0038] Figure 1 This is a schematic diagram of an online meeting scenario provided in an embodiment of this application;

[0039] Figure 2 This is a flowchart illustrating a meeting participation method provided in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of a menu displayed on a first terminal device according to an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the screen-flicking operation provided in the embodiments of this application;

[0042] Figure 5 This is a schematic diagram of a menu displayed on a second terminal device according to an embodiment of this application;

[0043] Figure 6 This is a schematic diagram illustrating the interaction between a first terminal device and its peer device based on a distributed operating system, as provided in an embodiment of this application.

[0044] Figure 7 This is a schematic diagram illustrating how the functionality of some modules of a conferencing application can be transferred to multiple peer devices, as provided in an embodiment of this application.

[0045] Figure 8 This is an interactive flowchart of an online meeting provided in an embodiment of this application;

[0046] Figure 9 This is a flowchart illustrating another meeting participation method provided in an embodiment of this application;

[0047] Figure 10 This is a schematic diagram illustrating how a specified video window is transferred to a peer device for display, as provided in an embodiment of this application.

[0048] Figure 11 This is a schematic diagram of another menu displayed on a second terminal device according to an embodiment of this application;

[0049] Figure 12 This is a schematic diagram of the structure of a meeting participation device provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of another meeting participation device provided in an embodiment of this application;

[0051] Figure 14 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0052] To facilitate understanding of the technical solution of this application, a brief introduction to one application scenario involved in this application will be given below.

[0053] Please see Figure 1 , Figure 1 This is a schematic diagram of an online meeting scenario provided in an embodiment of this application. The meeting server has a communication connection with the terminal device in the first physical space, which can be a wired connection or a wireless connection. The meeting server also has a communication connection with the terminal device in the second network space, which can be a wired connection or a wireless connection. This application does not make any specific limitations.

[0054] Figure 1 The conference server in the context of the application is responsible for conference coordination, acting as a communication bridge between different users (or participants) in the same online conference, forwarding relevant conference data (including video data, audio data, conference control information, etc.) between them. Regarding the type of conference server, it can be a selective forwarding unit or other media server; this application does not specify a particular type. The conference server can be deployed on a single computing device, a cluster of multiple computing devices, or a terminal device; this application also does not limit this deployment. The computing device can be a physical server, a virtual machine, a container, or an edge computing device. A virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity is used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system (BIOS), hard drive, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process. Edge computing devices refer to devices that are closer to the data source and end users, and feature low latency and high bandwidth, such as smart routers and edge servers. End devices can be desktop computers, laptops, tablets, etc., and this application does not limit them.

[0055] Figure 1 The first physical space includes one or more terminal devices. Users in the first physical space can reserve and / or participate in meetings through a conferencing application (APP) on the terminal devices in the first physical space. This application does not specifically limit the type and function of the conferencing APP. Figure 1The second physical space includes one or more terminal devices. Users in the second physical space can also reserve meetings and / or participate in meetings through the meeting app on the terminal devices in the second physical space. The first and second physical spaces mentioned above can be the same or different physical spaces. Regarding the type of physical space, it can be an office, bedroom, classroom, living room, conference hall, exhibition hall, game room, stage, square, gym, etc., and this application does not make specific limitations in this regard.

[0056] For example, such as Figure 1 As shown, assume physical space 1 is a living room containing terminal devices such as mobile phones, tablets, and laptops, and physical space 2 is an office containing terminal devices such as smart screens, mobile phones, and speakers. User 1 in physical space 1 establishes a communication connection with the conference server through a terminal device in physical space 1 to participate in the online conference with conference number 12345. User 2 in physical space 2 establishes a communication connection with the conference server through a terminal device in physical space 2, and thus participates in the online conference with conference number 12345. Since User 1 and User 2 are participating in the same online conference, they can exchange conference data.

[0057] It should be noted that, Figure 1 The physical spaces, terminal devices, and the number and types of users shown are merely examples and do not constitute specific limitations. In practical application scenarios, there can be more physical spaces. Users in different physical spaces can establish communication connections with the conference server through their terminal devices in their respective physical spaces, and thus participate in the corresponding online conferences. Different users in the same physical space can also participate in online conferences through different terminal devices in that physical space. This application does not impose specific limitations. Regarding the types of terminal devices in the physical space, they can be laptops, tablets, smartphones, smart screens, wearable devices (such as smartwatches, smart glasses, etc.), televisions, speakers, Bluetooth headsets, etc., and this application does not impose specific limitations.

[0058] Based on the preceding text Figure 1 The following describes an embodiment of the meeting participation method provided in this application.

[0059] Please see Figure 2 , Figure 2 This is a flowchart illustrating a meeting participation method provided in an embodiment of this application, including steps S201 to S205.

[0060] S201. In response to the first user's operation of entering the first conference on the first terminal device, the first terminal device sends the first media parameters to the conference server.

[0061] The aforementioned first terminal device belongs to the first user, meaning the first user has the right to use the first terminal device to participate in online meetings. The first user and the first terminal device are located in the same physical space. For details, please refer to [link / reference needed]. Figure 1 The relevant details are not elaborated here. The first meeting is a specific online meeting, and it has a corresponding identifier (ID). For example, it can be identified by a specific meeting number. The first meeting may also have a corresponding meeting password, meeting time, meeting topic, etc. This application does not impose specific limitations on the first meeting. The first user is a participant in the first meeting, and the first meeting may also have other participants.

[0062] The first terminal device has a conferencing application deployed on it, and the first user joins the first meeting through this application. This application does not specify the style or type of the conferencing application. A conferencing application is an application that provides online conferencing services to users. It includes multiple modules (software function modules) with different functions to support user participation in online meetings. This application does not limit the number of these modules or the specific function of each module.

[0063] For example, suppose a conferencing application includes a video capture and encoding module, a video decoding and rendering module, an audio capture and encoding module, an audio decoding and playback module, a conferencing collaboration module, a conferencing control module, and an artificial intelligence (AI) processing module. These modules have different functions and each module requires corresponding hardware support, which will be introduced separately below.

[0064] (1) Video capture and encoding module: This module is used to capture video data, encode the video data, and then send the encoded video data to the conference server. Running the video capture module on the terminal device requires hardware support from the terminal device, including a camera (for capturing video data), a processor (for video encoding), and a Network Interface Card (NIC) (also known as a network card / network adapter, used here to transmit the encoded video data to the conference server). The processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated hardware encoder, etc. This application does not specifically limit the type of camera, processor, video encoding method (the video encoding protocol used), or NIC type.

[0065] (2) Video Decoding and Rendering Module: This module decodes the encoded video data and renders the decoded video data for display. Running the video decoding and rendering module requires hardware support from the terminal device, including a processor (for video decoding) and a screen (for video rendering and display). This application does not specifically limit the video decoding method (the video decoding protocol used). Regarding the processor type, please refer to the previous description; it will not be repeated here. In addition to video decoding and rendering, the video decoding and rendering module can also provide video window selection functionality. For example, the video decoding and rendering module can render the video data of multiple users in the first conference on the terminal device's screen, with each user's video data corresponding to a video window. The video decoding and rendering module can also provide the function of zooming in, zooming out, or closing the video windows of certain users; this application does not specifically limit this functionality.

[0066] (3) Audio Acquisition and Encoding Module: This module is used to acquire and encode audio data, and then send the encoded audio data to the conference server. Running the audio acquisition and encoding module requires hardware support from the terminal device, such as a microphone (for acquiring audio data), a processor (for audio encoding), and a network card. This application does not specifically limit the type of processor, the audio encoding method (the audio encoding protocol used), the type of microphone, or the type of network card.

[0067] (4) Audio decoding and playback module: This module is used to decode the encoded audio data and play the decoded audio data. Running the audio decoding and playback module requires hardware support from the terminal device, such as a processor (for audio decoding) and a speaker (for playing audio). This application does not specifically limit the type of processor, the audio decoding method (the audio decoding protocol used), or the type of speaker.

[0068] (5) Meeting Collaboration Module: This module provides meeting collaboration functions, such as viewing shared content (including the desktop (the entire screen), whiteboard, application windows, documents, etc.), adding annotations and drawing graphics, remote control, and inputting and viewing chat content to improve meeting efficiency. Running the meeting collaboration module requires hardware support from the terminal device, such as a processor, screen, and network card. For example, the screen can capture content from the current terminal device that needs to be shared to other devices; the processor can encode the captured shared content; the network card can send the encoded shared content to the meeting server; the processor can also decode the encoded shared content from other devices; and the screen can render and display the shared content from other devices. This application does not specifically limit the type of processor and screen.

[0069] (6) Conference Control Module: This module provides conference control functions, such as controlling conference volume, controlling the speaking rights of users participating in the same conference, and controlling content sharing rights. Controlling a user's speaking rights can be achieved by muting or unmuteing them. When a user is muted, they have no speaking rights in the conference, and their audio data will not be collected and played on other users' devices. When a user is unmuteed, they have speaking rights in the conference, and their audio data can be collected and played on other users' devices. Running the conference control module requires hardware support from the terminal device, including input devices (such as touchscreens, mice, keyboards), processors, and network cards. Input devices are used for users to input conference control information (such as specifying conference volume and controlling user speaking rights). The processor encodes the conference control information input by users on this device. The network card transmits the encoded conference control information to the conference server via the network. The processor also decodes encoded conference control information from other devices and then performs corresponding conference control on this device based on the decoded information.

[0070] (7) AI Processing Module: This module provides AI processing functions, such as background replacement (which can replace the real background in video data with a virtual background), video beautification, voiceprint recognition, and automatic speech recognition (ASR). Running the AI ​​processing module requires hardware support such as a processor on the terminal device. This application does not specifically limit the type of processor; it can be a CPU, GPU, or AI-specific chip, etc.

[0071] It should be noted that the number of modules and the specific functions of each module in the above examples of meeting applications are merely illustrative and do not constitute a limitation. In practical applications, the modules in the above meeting applications can be further divided into multiple modules with different functions, or at least two modules can be merged into one module, or other modules can be added to the meeting application. This application does not impose any specific limitations. For ease of description, the following text will use the module division method of the meeting application given in the above examples for illustration, but it does not mean that meeting applications can only be divided into modules in this way.

[0072] Optionally, the multiple modules in a conferencing application can be pre-defined during the application development phase. This means the developers designed the application as a series of modules, so the conferencing application downloaded from an app store or official website directly contains these modules, without requiring the device to manually split the application into modules. For example, assuming the modules use the Harmony Ability Package (HAP) file format, where an HAP package is a module package generated from code, resources, third-party libraries, configuration files, etc., the conferencing application could include multiple HAP packages with different functionalities.

[0073] Optionally, the multiple modules in a conferencing application are actively split by the terminal device when downloading the application. For example, if the developer did not design the conferencing application as modules, the application downloaded by the terminal device from an app store or the application's official website may not directly contain multiple modules. The terminal device can dynamically split the conferencing application according to its functions, that is, decompose the conferencing application into multiple relatively independent modules, in which case the conferencing application includes these multiple modules.

[0074] Based on the first terminal device and the conferencing application deployed on it as described above, the first user can enter the first meeting on the first terminal device. This application does not specifically limit the first user's operation of entering the first meeting on the first terminal device. For example, the first terminal device runs the conferencing application, displaying the user interface of the conferencing application on its screen. The first user can then enter the meeting ID, meeting password, and their own user ID in the meeting application's user interface to enter the first meeting. This application also does not specifically limit the user ID; for example, a specific user number, username, or account can be used as the user ID to identify the corresponding user.

[0075] In response to a first user's entry into the first conference on a first terminal device, the first terminal device sends first media parameters to the conference server. The first media parameters indicate the data transmission method between the first user and the conference server; that is, the first media parameters specify how conference data for the first conference is transmitted between the first user and the conference server.

[0076] It should be understood that in order for a first user to participate in an online meeting, it is necessary to negotiate the method of transmitting meeting data between the first user and the meeting server. This method can be represented by some media parameters, including the address of the terminal device (which may be one or more devices) used by the first user to participate in the meeting, the corresponding ports of the various modules used to process the meeting data on the terminal device, the transmission protocol and bandwidth used to transmit the meeting data, the corresponding codec protocol type of the meeting data, and the resolution of the video data (a type of meeting data). Once these media parameters are negotiated, the terminal device used by the first user to participate in the meeting and the meeting server will follow these media parameters to transmit the meeting data of the online meeting. Since the first user in step S201 performs the operation of entering the first conference on the first terminal device, and the online conference that the first user wants to participate in is specifically the first conference, and the only device the first user is currently using to participate in the conference is the first terminal device, the first terminal device needs to negotiate with the conference server on the method by which the first user transmits the conference data of the first conference between the first terminal device and the conference server. At this time, the first media parameters sent by the first terminal device to the conference server include the encoding and decoding protocol type, resolution, bandwidth, transmission protocol type, Internet Protocol (IP) address of the first terminal device, and the corresponding ports of each module in the conference application on the first terminal device. The following is an explanation of each parameter.

[0077] The encoding / decoding protocol type in the first media parameter is related to the type of conference data, and this application does not impose specific limitations on it. For example, if the conference data of the first conference includes video data, then the encoding / decoding protocol here includes a video encoding protocol type and a video decoding protocol type to indicate how the video data transmitted between the first terminal device and the conference server requires video encoding, and to indicate how the first terminal device should decode the video data received from the conference server. If the conference data includes audio data, then the encoding / decoding protocol here includes an audio encoding protocol type and an audio decoding protocol type to indicate how the audio data transmitted between the first terminal device and the conference server requires audio encoding, and to indicate how the first terminal device should decode the audio data received from the conference server.

[0078] The resolution in the first media parameter is used to indicate the resolution required for the video data transmitted between the first terminal device and the conference server. This application does not specifically limit the value of this resolution.

[0079] The bandwidth in the first media parameter is used to indicate the bandwidth used when transmitting conference data between the first terminal device and the conference server.

[0080] The transmission protocol type in the first media parameter is used to indicate the transmission protocol used when transmitting conference data between the first terminal device and the conference server. This application does not specifically limit the transmission protocol here.

[0081] The IP address of the first terminal device and the corresponding ports of each module in the conferencing application on the first terminal device in the first media parameters are used to instruct the conferencing server how to send the conferencing data of the first meeting to the first user, and also to help the conferencing server determine whether the received conferencing data of the first meeting comes from the first user. The ports corresponding to different modules in the conferencing application on the first terminal device may be the same or different, depending on the design of the conferencing application, and this application does not make specific limitations.

[0082] For example, suppose the conferencing application deployed on the first terminal device includes a video capture and encoding module and a video decoding and rendering module. The first media parameters sent by the first terminal device to the conferencing server include the IP address of the first terminal device and port 1 corresponding to the two modules on the first terminal device. Correspondingly, the conferencing server records the correspondence between the first user and the IP address and port 1 of the first terminal device. When the conferencing server receives video data, it determines whether the IP address and port carried by the video data match the IP address and port 1 of the first terminal device in the aforementioned correspondence. If they match, the conferencing server determines that the video data originates from the first user and forwards it to the devices of other participants in the same meeting. When the conferencing server receives video data from other participants, it can determine how to send the video data to the first user based on the aforementioned correspondence. That is, it uses the IP address of the first terminal device in the aforementioned correspondence as the destination IP address and port 1 as the destination port, thus forwarding the conferencing data to the first user.

[0083] For example, suppose a conferencing application includes a video capture and encoding module, a video decoding and rendering module, an audio capture and encoding module, an audio decoding and playback module, a conferencing collaboration module, and a conferencing control module. The first five modules process conferencing data including audio data, video data, and shared content (documents, application windows, desktop, etc.), all of which are multimedia data. These five modules can correspond to the same port on the first terminal device, denoted as the media stream port. The conferencing control module processes conferencing control information, and the conferencing control module corresponds to a different port on the first terminal device (different from the aforementioned media stream port).

[0084] S202, The conference server records the correspondence between the first conference, the first user, and the first media parameters, and sends the confirmation result of the first media parameters to the first terminal device.

[0085] For ease of description, the above correspondence can be represented as First Meeting - First User - First Media Parameter. This means that the First User is one of the participants in the First Meeting, and the transmission of meeting data between the First User (the terminal device used) and the meeting server must follow the First Media Parameter. That is, the First Media Parameter indicates the data transmission method between the First User and the meeting server in the First Meeting.

[0086] The above confirmation result indicates that the conference server has determined (agreed / allowed) to transmit the conference data of the first conference between the first user and the conference server in accordance with the requirements of the first media parameters.

[0087] It should be understood that through steps S201 and S202, the first terminal device and the conference server complete the negotiation of the first media parameters. Then, the first terminal device currently used by the first user to participate in the meeting and the conference server will transmit the meeting data of the first meeting in accordance with the requirements of the first media parameters, so that the first user can participate in the first meeting on the first terminal device and enjoy the online meeting service provided by the meeting application on the first terminal device.

[0088] S203. The first terminal device obtains a first instruction. The first instruction instructs a second terminal device to replace the first module in the conferencing application of the first terminal device to process the first data. The first data is all conferencing data related to the function of the first module in the first conference.

[0089] The aforementioned first terminal device and second terminal device reside in the same physical space, and both belong to a first user within that physical space. That is, the first user has the right to use both the first and second terminal devices, and therefore can participate in online meetings through the first terminal device and / or the second terminal device. The first and second terminal devices can be of the same or different types. This application does not specifically limit the type of terminal device or its physical space; please refer to relevant documentation. Figure 1 The relevant information is omitted here.

[0090] The second terminal device is the peer device of the first terminal device. The peer device of the first terminal device refers to a terminal device that has a communication connection with the first terminal device and supports running one or more modules of the conferencing application. It should be understood that for a terminal device to support running a module of the conferencing application, it means that the terminal device has the hardware and software environment (operating system) required for that module to run. In addition to the second terminal device, the first terminal device may have more peer devices; this application does not limit this.

[0091] The first terminal device can determine which peer devices it has through device discovery technology.

[0092] Specifically, the first terminal device can broadcast a device discovery message, which is used to search for peer devices. Other terminal devices with a communication connection to the first terminal device can receive this device discovery message. For example, other terminal devices in the same Wireless Local Area Network (WLAN) or other networks as the first terminal device can receive the message, as can other terminal devices within the Bluetooth signal coverage area of ​​the first terminal device. This application does not impose specific limitations on this. When other terminal devices receive the device discovery message sent by the first terminal device, they can send their own device information to the first terminal device. This device information is used to indicate which / which modules in the conferencing application the corresponding terminal device supports running.

[0093] This application does not limit the specific content of the aforementioned device information. For example, device information can be the hardware information of the terminal device (such as the type and model of the hardware contained in the terminal device) and the software environment information (such as the type and version of the operating system). Through the hardware information and software environment information of a terminal device, it can be determined which / which modules in the conferencing application the terminal device supports (or it is possible that no module is supported).

[0094] Alternatively, the first terminal device can send a functional coordination message to indicate the modules in the conferencing application on the first terminal device. Other terminal devices with a communication connection to the first terminal device can receive this functional coordination message. For example, other terminal devices in the same WLAN network or other networks as the first terminal device can receive this functional coordination message, as can other terminal devices within the Bluetooth signal coverage range of the first terminal device. This application does not make specific limitations. When other terminal devices receive the functional coordination message sent by the first terminal device, they can determine which / which modules they can support running based on their own hardware and software environment (it is also possible that no modules are supported). If a terminal device finds that it does not support running any module in the conferencing application, then that terminal device is not the peer device of the first terminal device. If a terminal device finds that it can support running one or more modules in the conferencing application, then that terminal device can determine that it is the peer device of the first terminal device and then report back to the first terminal device.

[0095] This application does not specify a particular time for device discovery, as long as it occurs before step S203. For example, device discovery can be performed when the first terminal device starts up, allowing the first terminal device to determine which peer devices it has. Alternatively, device discovery can be performed when the first terminal device is running a conferencing application, and so on.

[0096] The first module in step S203 belongs to one or more modules supported by the second terminal device. The first module can be any or a specific module among multiple modules in the conferencing application (the first module may include one or more modules). The type of conferencing data processed by the first module is related to the function of the first module. This application does not limit the specific function of the first module. For example, assuming the first module is the video acquisition and encoding module described above, the first module is used to acquire video data and encode the video data, and then send the encoded video data (belonging to conferencing data) to the conferencing server. As another example, assuming the first module is the video decoding and rendering module described above, the first module is used to decode and render the video data sent from the conferencing server. As yet another example, assuming the first module is the audio acquisition and encoding module described above, the first module is used to acquire audio data and encode the audio data (belonging to conferencing data), and then send the encoded audio data to the conferencing server.

[0097] This application does not specify the method for obtaining the first instruction mentioned above. The following explanation will use method one to method two as examples.

[0098] Method 1: In response to the first user's first operation on the first terminal device, the first terminal device generates a first instruction.

[0099] In this method, the first terminal device automatically generates a first instruction based on a first operation performed by a first user on the first terminal device. The first operation includes instructing the first module to perform an operation and instructing the second terminal device to perform an operation. This application does not specifically limit the type of the first operation; the following explanation uses operation methods 1 to 2 as examples.

[0100] Operation Method 1: The first user selects the hardware and a second terminal device required for the first module to run on the first terminal device. Accordingly, the first terminal device can be mapped to the first module based on the hardware selected by the first user, and then, in conjunction with the second terminal device selected by the first user, the first terminal device automatically generates the first instruction.

[0101] For example, Figure 3 A menu displayed on the first terminal device is described, including a primary menu and secondary menus corresponding to each option within the primary menu. The primary menu lists the hardware corresponding to each functional module in the conferencing application, including options such as camera, screen, microphone, speaker, and processor. The secondary menu corresponding to each hardware option includes the peer devices that support the first terminal device in running the module corresponding to that hardware option. Each secondary menu for a hardware option may have one or more peer device options, or it may have no peer device options (indicating that the first terminal device does not have a peer device capable of supporting the module corresponding to that hardware option).

[0102] Suppose the camera on the first terminal device has a low resolution, and the first user, while participating in a first meeting through the first terminal device, wants to use the camera on the other end of the first terminal device instead of the first terminal device's camera to improve the meeting experience. For example... Figure 3 As shown, the first user can select the camera option from the primary menu on the first terminal device. As previously described, the hardware required for the video capture and encoding module to operate includes a camera and a processor. Therefore, when the first user selects the camera option in the primary menu, the first terminal device can determine the video capture and encoding module based on the mapping relationship between the camera and the module. Then, the first terminal device further displays a secondary menu corresponding to the camera option. This secondary menu lists two peer devices: a second terminal device and a third terminal device, both of which support the video capture and encoding module. Assuming the second terminal device's camera has a higher resolution than the first terminal device's, the first user can select the second terminal device option in the secondary menu. Combined with the user's previous operation, the first terminal device automatically generates a first instruction. This instruction instructs the second terminal device to replace the video capture and encoding module (corresponding to the first module described earlier) in the first conference to process the conference data related to the video capture and encoding module's function, i.e., the video data in the first conference.

[0103] It should be noted that the first menu in the above example lists hardware options, and the module corresponding to the hardware option selected by the user is the first module. In practical scenarios, in addition to indirectly selecting the first module through hardware options, the identifiers of each module in the conferencing application can also be listed in the first-level menu. The module identifier can be the module's name, function description, number, or icon, etc. This application does not make specific limitations on this. Then, the first terminal device can map the identifier selected by the user to the corresponding module, and thus determine that module as the first module.

[0104] Operation Method 2: The first user performs the motion-sensing operation corresponding to the first module and the motion-sensing operation corresponding to the second module on the first terminal device.

[0105] Regarding the type of motion-sensing operation, it can be screen-shaking interaction, detection of specific gestures, etc., and this application does not impose specific limitations. Users can also customize motion-sensing operations, such as setting different motion-sensing operations to correspond to different modules in the meeting application, setting different motion-sensing operations to correspond to different peer devices, and so on. The motion-sensing operation corresponding to the first module and the motion-sensing operation corresponding to the second module can be the same or different motion-sensing operations. For example, motion-sensing operation 1 corresponds to the first module, and motion-sensing operation 2 corresponds to the second terminal device. The first user needs to execute motion-sensing operation 1 and motion-sensing operation 2 on the first terminal device to specify the first module and the second terminal device. Alternatively, motion-sensing operation 3 corresponds to the first module and the second terminal device. The first user can specify the first module and the second terminal device by executing motion-sensing operation 3 on the first terminal device.

[0106] For example, such as Figure 4 As shown in the figure, assuming the mobile phone in the figure is the first terminal device, the laptop and smart screen in the figure are the mobile phone's counterpart devices, and the first user can specify the first module and the second terminal device by performing specific motion-sensing operations.

[0107] If the first user wishes to use a laptop to replace the video decoding and rendering module in the mobile phone to process the encoded video data (corresponding to the first data described above), the first user can bring the mobile phone close to the laptop and then shake the phone once towards the laptop (i.e., screen-shaking operation 1). The mobile phone detects screen-shaking operation 1, which corresponds to the laptop and the video decoding and rendering module. Therefore, the mobile phone identifies the video decoding and rendering module in the phone as the first module and the laptop as the second terminal device, thereby generating a corresponding first instruction. This first instruction instructs the laptop to replace the video decoding and rendering module in the mobile phone to process the encoded video data (i.e., the meeting data related to the function of the video decoding and rendering module) so that the corresponding video content can be displayed on the laptop.

[0108] If the first user wants to use the smart screen to replace the video capture and encoding module in their mobile phone to process video data, the user can bring their phone close to the smart screen and then shake the phone twice in the direction of the smart screen (i.e., screen-shaking operation 2). The phone detects screen-shaking operation 2, which corresponds to the smart screen and the video capture and encoding module. Therefore, the phone identifies the video capture and encoding module as the first module and the smart screen as the second terminal device, thus generating a corresponding first instruction. This first instruction instructs the user to use the smart screen to replace the video capture and encoding module in their mobile phone to process video data (i.e., meeting data related to the function of the video capture and encoding module).

[0109] It should be noted that the screen-shaking operation described in the above examples is only an example and does not constitute a limitation. In actual application scenarios, there may be other motion-sensing operations, and this application does not make any specific limitations.

[0110] Method 2: In response to the second operation of the first user on the second terminal device, the second terminal device sends a first instruction to the first terminal device.

[0111] Unlike Method 1, where the first terminal device actively generates the first instruction, in Method 2, the first terminal device does not actively generate the first instruction. Instead, it receives the first instruction from the second terminal device. The first instruction is generated by the second terminal device based on a second operation performed by the first user on the second terminal device. The second operation includes instructing the first module to perform an operation. This application does not specifically limit the operation instructing the first module.

[0112] For example, Figure 5 A menu displayed on the second terminal device is provided, which lists the hardware corresponding to the modules that the second terminal device supports running, including options such as camera, screen, microphone, speaker, processor, etc., and one or more modules in the conferencing application corresponding to each hardware option.

[0113] Assume the first module is the video capture and encoding module in the first terminal device. The hardware corresponding to this module includes a camera and a processor. The first user wants to transfer the functionality of the video capture and encoding module from the first terminal device to a second terminal device. Therefore, the first user can select the "camera" option from the menu displayed on the second terminal device. Responding to this operation, the second terminal device identifies the camera and determines its corresponding video capture and encoding module as the first module, generating a first instruction. This first instruction instructs the second terminal device to replace the video capture and encoding module in the first terminal device to process the video data (i.e., meeting data related to the video capture and encoding module's functionality) for the first user in the first meeting. Then, the second terminal device sends the first instruction back to the first terminal device.

[0114] S204. The first terminal device sends a second instruction to the second terminal device according to the first instruction. The second instruction instructs the second terminal device to run a second module within the second terminal device to process the first data. The second module has the same function as the first module. The first data processed by the second terminal device is either sent to the second terminal device by the first terminal device / conference server or collected by the second terminal device.

[0115] As described in step S203, the second terminal device is the peer device of the first terminal device, and the first instruction received by the first terminal device instructs the second terminal device to replace the first module in the first terminal device to process the first data. The first data is all conference data related to the function of the first module in the first conference. To achieve the same function as the first module on the second terminal device (fulfilling the requirements of the first instruction), a module with the same function as the first module needs to be installed on the second terminal device, namely the aforementioned second module. The second module has the same function as the first module. The second terminal device runs the second module, and then the second module on the second terminal device can replace the first module on the first terminal device to process the first data for the first user.

[0116] Optionally, the second module can be pre-installed in the second terminal device. That is, when the first terminal device obtains the first instruction, the first terminal device has already determined that the second terminal device has a second module with the same function as the first module. Therefore, the second terminal device does not need to install a module with the same function as the first module. Then, the first terminal device sends a second instruction to the second terminal device according to the first instruction to instruct the second terminal device to run the existing second module.

[0117] Optionally, the second module can be derived from the first module. That is, when the first terminal device receives the first instruction, it determines that the second terminal device does not yet have a second module with the same function as the first module. Therefore, the second terminal device needs to install a second module with the same function as the first module. In this case, the first terminal device can copy the first module from its own device to the second terminal device. This copying essentially involves installing a module with the same function as the first module in the second terminal device, thus obtaining the second module in the second terminal device. Alternatively, the first terminal device can instruct the second terminal device to download and install a module with the same function as the first module. Accordingly, the second terminal device can download and install the same module from an app store or app website, thus obtaining the second module in the second terminal device. Then, the first terminal device sends a second instruction to the second terminal device according to the first instruction, instructing the second terminal device to run the second module.

[0118] For example, such as Figure 6As shown, a distributed operating system (or distributed soft bus) is deployed on the first terminal device and multiple peer devices (denoted as peer devices 1 to N, including the second terminal device). This distributed operating system spans the first terminal device and these peer devices, essentially establishing a communication bridge (providing a transmission channel) between them, supporting the copying of modules from the conferencing application on the first terminal device to any peer device. Therefore, when the first terminal device receives a first instruction, it can copy the first module from the first terminal device to the second terminal device through this distributed operating system, thereby obtaining the second module in the second terminal device. Regarding the type of the aforementioned distributed operating system, it can be HarmonyOS or other distributed operating systems; this application does not specifically limit its application.

[0119] It should be noted that, Figure 6 The modules in the conferencing application on the first terminal device and the modules on the peer device shown are merely examples and do not constitute a limitation. In real-world scenarios, conferencing applications can have other module division methods. One or more modules from the first terminal device can be copied to one or more peer devices through a distributed operating system. The number and type of modules on each peer device are not limited, and users can choose according to their needs.

[0120] It should be understood that when the second terminal device runs the second module to replace the first module in the first terminal device to perform the corresponding functions, the second module may need to transmit data with the conference server. The data transmission direction can be uplink, that is, from the second terminal device to the conference server to transmit the first data processed by the second module, or downlink, that is, from the conference server to the second terminal device to transmit the first data that the second module needs to process.

[0121] The data transmission direction described above depends on the function of the second module (i.e., the function of the first module). For example, assuming the second module is a video capture and encoding module, this module needs to encode the video data (corresponding to the first data) captured by the device's camera, and then send the encoded video data (corresponding to the processed first data) to the conference server. In this case, the data transmission direction is upstream. Alternatively, assuming the second module is a video decoding and playback module, this module needs to receive the encoded video data (corresponding to the first data) sent by the conference server, and then decode and render it on the device. In this case, the data transmission direction is downstream.

[0122] Optionally, the second module in the second terminal device can transmit data with the conference server through the first terminal device, or it can transmit data directly with the conference server (i.e., without forwarding through the first terminal device). The following describes the scenarios.

[0123] Scenario 1: The data transmission direction between the second module and the conference server is uplink, and the second module in the second terminal device transmits data with the conference server through the first terminal device.

[0124] In this scenario, the first terminal device needs to record the mapping relationship between the first port corresponding to the first module in the first terminal device and the second port corresponding to the second module in the second terminal device. Subsequently, the second terminal device can send the processed first data to the first terminal device through the IP address of the second terminal device + the second port (as the source IP and source port). Then, the first terminal device sends the processed first data to the conference server through the IP address of the first terminal device + the first port (as the modified source IP and source port) according to the above mapping relationship.

[0125] Scenario 2: The data transmission direction between the second module and the conference server is downlink, and the second module in the second terminal device transmits data with the conference server through the first terminal device.

[0126] In this scenario, the first terminal device needs to record the mapping relationship between the first port corresponding to the first module in the first terminal device and the second port corresponding to the second module in the second terminal device. Subsequently, when the first terminal device receives the first data sent by the conference server, the destination IP address and destination port of the first data are the IP address of the first terminal device + the first port. Based on the above mapping relationship, the first terminal device changes the destination IP address and destination port of the first data to the IP address of the second terminal device + the second port, and then forwards the first data to the second terminal device.

[0127] It should be noted that since the second module in the second terminal device in both scenarios 1 and 2 transmits data between the first terminal device and the conference server, the conference server will not be aware that the second module in the second terminal device has replaced the first module in the first terminal device. Therefore, it does not need to modify the correspondence between the first conference, the first user, and the first media parameters recorded on the conference server (including the correspondence between the IP address of the first user and the first terminal device and the port corresponding to the first module on the first terminal device). The conference server and the first terminal device will still transmit data according to the requirements of the first media parameters in step S202. For details, please refer to the relevant introduction in step S202, which will not be repeated here.

[0128] Scenario 3: The data transmission direction between the second module and the conference server is uplink, and the second module in the second terminal device directly transmits data with the conference server.

[0129] In this scenario, since the first data collected and processed by the second module is sent from the second terminal device to the conference server without being forwarded by the first terminal device, it is necessary to modify the correspondence between the first user and the IP address of the first terminal device and the first port corresponding to the first module recorded on the conference server in step S202 to the correspondence between the first user and the IP address of the second terminal device and the second port corresponding to the second module. This will allow the conference server to determine that the conference data is from the first user based on the IP address and second port of the second terminal device in the received conference data, and then forward the first user's conference data to other users participating in the same conference.

[0130] Scenario 4: The data transmission direction between the second module and the conference server is downlink, and the second module in the second terminal device directly transmits data with the conference server.

[0131] In this scenario, since the conference server needs to send the first data to the second module in the second terminal device for processing without forwarding through the first terminal device, the mapping relationship between the first user and the IP address of the first terminal device and the first port corresponding to the first module recorded on the conference server in step S202 needs to be modified to the mapping relationship between the first user and the IP address of the second terminal device and the second port corresponding to the second module, so that the conference server can send the first data to the second module in the second terminal device according to the IP address of the second terminal device + the second port (as the destination IP and destination port).

[0132] It should be noted that, since the second module in the second terminal device in scenarios three and four interacts directly with the conference server without going through the first terminal device, it is necessary to modify the corresponding records on the conference server. The specific process is explained below.

[0133] After the first terminal device receives the first instruction, it can send parameter update information to the conference server. This parameter update information includes the IP address of the second terminal device and the port (i.e., the second port) corresponding to the second module on the second terminal device. As described in step S202, the conference server records a first correspondence, which is the correspondence between the addresses of the first user and the first terminal device and the first port corresponding to the first module on the first terminal device. Before the conference server receives the parameter update information, the transmission of conference data related to the first module between the first user and the conference server follows this first correspondence. That is, the conference server can send corresponding conference data to the first user based on the address and first port of the first terminal device, or confirm whether the received conference data originates from the first user based on the address and first port of the first terminal device.

[0134] After receiving the aforementioned parameter update information, the conference server changes the previously recorded first correspondence to a second correspondence between the IP address and second port of the first user and the second terminal device, and then sends a confirmation message to the first terminal device. The confirmation message indicates that the conference server has changed the recorded first correspondence to the second correspondence, signifying that the first terminal device has renegotiated the method for transmitting the first data (i.e., conference data related to the function of the first module) between the first user and the conference server. Subsequently, the conference server will either send the first data to the first user based on the IP address and second port of the second terminal device, or confirm whether the received processed first data originates from the first user based on the IP address and second port of the second terminal device.

[0135] The IP address of the second terminal device and the second port corresponding to the second module are obtained by the first terminal device from the second terminal device before sending the above parameter update information. This application does not specify the time when the first terminal device obtains the IP address and second port of the second terminal device.

[0136] For example, when the first user performs the first operation on the first terminal device (refer to method one in step S203), the first terminal device generates a first instruction. Based on this first instruction, the first terminal device can obtain the IP address of the second terminal device and the second port corresponding to the second module from the second terminal device. Then, it carries the IP address of the second terminal device and the second port corresponding to the second module in the parameter update information and sends it to the conference server. Alternatively, the first terminal device can obtain the IP address of the second terminal device during the device discovery process (i.e., discovering the peer device), and then, when generating the first instruction, obtain the second port corresponding to the second module on the second terminal device from the second terminal device, and then send the parameter update information carrying the IP address and second port of the second terminal device to the conference server.

[0137] For example, when the first user performs the second operation on the second terminal device (refer to method two in step S203), the second terminal device generates a first instruction and then sends the first instruction to the first terminal device. At this time, the second terminal device can also send the IP address of the second terminal device and the second port corresponding to the second module to the first terminal device, so that the first terminal device can carry the IP address of the second terminal device and the second port corresponding to the second module in the parameter update information in step S204 to send to the conference server.

[0138] Optionally, when the first terminal device receives the aforementioned confirmation information sent by the conference server, it indicates that the conference server and the first user have completed renegotiation. The first module in the first terminal device will be replaced by the second module in the second terminal device. Thus, the first terminal device can stop running the first module to reduce the processor load, power consumption, and heat generation of the device.

[0139] Optionally, after the first terminal device receives the confirmation information sent by the conference server, the first terminal device may send the parameters related to the first module in the first media parameters to the second terminal device. The parameters related to the first module include the codec protocol type, resolution, bandwidth, transmission protocol type, IP address and port of the conference server.

[0140] The parameters related to the first module are determined by its function. For example, assuming the first module is a video capture and encoding module, the parameters related to this module include the video encoding protocol type, resolution, bandwidth, transmission protocol type, and the IP address and port of the conference server. The first terminal device can send these parameters to the second terminal device so that the video capture and encoding module in the second terminal device can transmit data with the conference server according to these parameters. Similarly, assuming the first module is an audio decoding and playback module, the parameters related to this module include the audio decoding protocol type, the IP address and port of the conference server. The first terminal device can send these parameters to the second terminal device so that the audio decoding and encoding module in the second terminal device can receive the encoded audio data sent by the conference server and perform audio decoding and playback according to the corresponding audio decoding protocol type.

[0141] S205. The second terminal device runs the second module to process the first data according to the second instruction.

[0142] The second instruction, the second module, and the first data can be found in the previous text and will not be repeated here.

[0143] It should be noted that, Figure 3 This describes replacing the first module on the first terminal device with a second terminal device to achieve the corresponding functions; that is, transferring the functionality of the first module in the conferencing application on the first terminal device to the second device. In fact, the first user can also refer to... Figure 3 The method involves transferring the functions of other modules in the conferencing application of the first terminal device, excluding the first module, to the second terminal device or other peer devices. For the sake of brevity, this will not be elaborated here.

[0144] For example, such as Figure 7As shown in the diagram, assuming the first terminal device is mobile phone A, the conferencing application on the first terminal device includes a video capture and encoding module, a video decoding and rendering module, a conferencing collaboration module, a conferencing control module, an audio capture and encoding module, and an audio decoding and playback module. The laptop, smart screen, mobile phone B in the lower left corner, and speaker in the diagram are all in the same physical space as mobile phone A; they are all peer devices of mobile phone A. Initially, the first user participates in the first meeting solely based on the conferencing application on mobile phone A. However, due to the limited capabilities of mobile phone A, such as a small screen, low camera resolution, and poor speaker performance, the first user's meeting experience is poor. To improve the first user's meeting experience in the current scenario, the first user can transfer the functionality of some modules of the conferencing application on mobile phone A to suitable peer devices (one or more peer devices replacing some modules on mobile phone A to implement the corresponding functions), thereby comprehensively utilizing the capabilities of multiple terminal devices (inter-device capability collaboration) to participate in the first meeting.

[0145] Specifically, due to Figure 7 The laptop's camera has a higher resolution than the phone's camera. The laptop's camera is also better positioned and oriented (capturing the user's full face, body, or the entire physical space). Furthermore, the laptop's screen is larger than the phone's. Therefore, user A can choose to transfer the video capture and encoding modules and the video decoding and rendering modules from the conferencing application on phone A to the laptop. For details on how to transfer these modules, please refer to [reference needed]. Figure 3 The details of the previous description are omitted here. Then, the laptop, based on its camera, processor, and other hardware, runs the video capture and encoding module on its device, allowing the first user to enjoy the functionality of this module: the laptop captures video data through its camera, encodes the video data using the processor described in this application, and then sends the encoded video data to the conference server, so that the conference server can forward the video data to the devices of other participants for rendering and display. The laptop also runs the video decoding and rendering module on its device, based on its processor and screen, allowing the first user to enjoy the functionality of this module: the laptop decodes the video data from other participants sent by the conference server and then renders and displays the decoded video data on its screen.

[0146] Because the smart screen has a larger screen and higher resolution than mobile phone A, and is more convenient for viewing and editing the shared content of the first meeting, user A can also choose to transfer the meeting collaboration module function from the meeting application on mobile phone A to the smart screen. Then, the smart screen runs the meeting collaboration module on its own device based on its screen, processor, and other hardware, allowing the first user to enjoy the functions of the meeting collaboration module on the laptop: the smart screen renders and displays the shared content through its own screen, and can also input meeting collaboration information through the screen (touchscreen), such as annotating the shared content, drawing patterns, and other editing operations on the screen. The smart screen can also collect the shared content and corresponding meeting collaboration information through the processor of this application, and then send the meeting collaboration information to the meeting server, so that the meeting server can forward the meeting collaboration information to the terminal devices of other participants for corresponding display.

[0147] Since the speaker in the speaker has better sound quality and louder volume than the speaker in phone A, the first user can transfer the audio decoding and playback module function from the meeting application in phone A to the speaker. The speaker runs the audio decoding and playback module on its own device based on the device's processor, speaker and other hardware, so that the first user can enjoy the function of the audio decoding and playback module through the speaker: the speaker decodes the audio data of other participants sent by the meeting server through the device's processor, and then plays the decoded audio data through the device's speaker.

[0148] Since phone B is closer to the first user and more convenient for meeting control, the first user can also transfer the functions of the meeting control module from phone A to phone B. Phone B runs the meeting control module on its own device based on its screen, processor, and other hardware, allowing the first user to enjoy the functions of the meeting control module on phone B: the screen of phone B displays the corresponding meeting control interface / window of the meeting control module, and can also collect the meeting control information entered by the user on the screen (touchscreen), such as controlling the speaking rights of participants (mute or unmute), controlling the meeting volume, etc. Then, the processor of phone B encodes the collected meeting control information, and then phone B sends the encoded meeting control information to the meeting server. Subsequently, the meeting server sends the meeting control information to the terminal devices of other participants to realize the corresponding meeting control.

[0149] After completing the functional transfer of the aforementioned modules, phone A retains only the audio acquisition and encoding module function within the conferencing application. Phone A utilizes its microphone and processor to run this module, acquiring and encoding audio data before sending it to the conferencing server. The server then forwards the audio data to other participants' devices for playback. Compared to participating in a meeting solely through phone A, this example spans multiple devices, allowing them to leverage each other's strengths (hardware capabilities working together) to provide users with a distributed and efficient conferencing experience.

[0150] It should be noted that, Figure 7 The number of devices, the number of modules requiring function transfer, and the location (peer device) to which each module is transferred are all examples and do not constitute specific limitations. In reality, phone A can transfer more or fewer modules from the conferencing application to the peer device. The number of peer devices selected can be more or less, and each peer device can implement the functions of one or more modules from the conferencing application. For example, the audio decoding and playback module function of phone A can be transferred to a laptop computer, or the video decoding and playback module function can be transferred to a tablet computer (also a peer device of phone A, not shown in the diagram). Alternatively, the conferencing collaboration module function can not be transferred to phone B, in which case phone A retains the conferencing collaboration module function, and the first user still conducts conferencing collaboration on phone A.

[0151] Next, let's combine... Figure 8 ,right Figure 2 Examples of the methods for participating in the meeting are provided.

[0152] Please see Figure 8 , Figure 8 This is an interactive flowchart of an online meeting provided in an embodiment of this application, including steps 1 to 20.

[0153] It should be noted that, Figure 8The first terminal device is located in the first physical space. A conferencing application is deployed on the first terminal device, which includes multiple modules with different functions (details can be found in the previous text and will not be repeated here). The first physical space includes the first terminal device and at least one peer device of the first terminal device. A first user in the first physical space can participate in an online meeting individually using the first terminal device, or can participate in an online meeting using both the first terminal device and its peer device. The second terminal device is one of the peer devices of the first terminal device and is located in the same physical space as the first terminal device. Terminal device X is located in the second physical space (different from the first physical space). A conferencing application is also deployed on terminal device X. A second user in the second physical space can participate in an online meeting individually using terminal device X, or can participate in an online meeting using both terminal device X and its peer device (also located in the second physical space).

[0154] Step 1. The first terminal device and the second terminal device perform device discovery.

[0155] Specifically, the first terminal device determines, based on device discovery technology, that the second terminal device is its peer device, meaning the second terminal device supports running at least one module of the conferencing application on the first terminal device. For details on the specific methods of device discovery, please refer to the relevant description in step S202; it will not be repeated here. For ease of description, the following explanation uses the example of the second terminal device supporting the video capture and encoding module in the conferencing application. In reality, the second terminal device can also support running other modules in the conferencing application.

[0156] Step 2. The first terminal device detects that the first user has entered the first conference.

[0157] Specifically, the first user can enter the first conference on the first terminal device. The specific operation method can be found in step S201, and will not be repeated here. In response to the first user's entry into the first conference, the first terminal device executes step 3, which involves negotiating the first user's media parameters with the conference server to determine how to transmit the conference data between the first user and the conference server.

[0158] Step 3. The first terminal device sends the first media parameters to the conference server.

[0159] For details regarding the first media parameter here, please refer to the description in step S202; it will not be repeated here.

[0160] Step 4. The conference server sends the confirmation result of the first media parameter to the first terminal device.

[0161] Specifically, when the conference server receives the first media parameters sent by the first terminal device, the conference server can record the correspondence between the first conference, the first user, and the first media parameters. Then, the conference server sends an acknowledgment result of the first media parameters to the first terminal device. This acknowledgment result indicates that the conference server confirms (agrees to) the transmission of conference data for the first conference between the first user and the conference server according to the first media parameters. The conference server can also send its IP address and port to the first terminal device.

[0162] Step 5. Terminal device X detects that the second user has entered the first meeting.

[0163] Specifically, the second user can enter the first conference on the second terminal device, meaning the second user and the first user enter the same conference, and they are different participants in the same conference. In response to the second user's entry into the first conference, the second terminal device executes step 6, which involves the terminal device X negotiating the corresponding media parameters of the second user with the conference server to determine how to transmit the conference data of the first conference between the second user and the conference server.

[0164] Step 6. Terminal device X sends the second media parameters to the conference server.

[0165] It should be understood that in order to participate in the first meeting, the second user needs to negotiate corresponding media parameters with the meeting server. These parameters include the address of the terminal device used by the second user to join the meeting, the ports corresponding to the various modules used to process meeting data on the terminal device, the codec protocol type, resolution, bandwidth, and transmission protocol type. This allows the second user's terminal device to know how to transmit meeting data with the meeting server. After negotiation, the second user and the meeting server will follow these media parameters to transmit the meeting data of the first meeting. In other words, the second media parameters are used to indicate the data transmission method between the second user and the meeting server. Since the second user is currently only using the second terminal device to participate in the first meeting, the second media parameters sent by the second terminal device to the meeting server in step 6 include the address of the second terminal device, the ports corresponding to the various modules in the meeting application on the second terminal device, the codec protocol type supported by the second terminal device, resolution, bandwidth, and transmission protocol type.

[0166] Step 7. The conference server sends the confirmation result of the second media parameters to the second terminal device.

[0167] Specifically, when the conference server receives the second media parameters sent by the second terminal device, it can record the correspondence between the first conference, the second user, and the second media parameters. This indicates that the second user is a participant in the first conference, and the transmission of conference data between the second user and the conference server must follow the second media parameters; that is, the second media parameters indicate the data transmission method between the second user and the conference server in the first conference. Then, the conference server sends an acknowledgment result of the second media parameters to the second terminal device. This acknowledgment result indicates that the conference server confirms (agrees to) the transmission of conference data between the second user and the conference server according to the second media parameters. The conference server can also send its IP address and port to the second terminal device.

[0168] It should be noted that, Figure 8 The order of steps 1-4 and steps 5-7 shown in the diagram is for illustrative purposes only and does not constitute a limitation. It is sufficient to ensure that the order of steps 1-4 remains unchanged, the order of steps 5-7 remains unchanged, and steps 1-7 occur before step 8. In other words, the first user and the second user can negotiate with the conference server to join the first conference simultaneously, or they can negotiate with the conference server to join the first conference sequentially.

[0169] Step 8. The first terminal device sends video data to the conference server.

[0170] Specifically, since the first terminal device has already negotiated the media parameters related to the first user with the conference server, namely the first media parameters, the first terminal device can collect the video data of the first user (which belongs to the conference data of the first conference) based on the video acquisition and encoding module in the conference application, encode the video data according to the encoding and decoding protocol type in the first media parameters, and then send the encoded video data to the conference server.

[0171] Step 9. The conference server sends video data to terminal device X.

[0172] Specifically, when the conference server receives video data (encoded) sent by the first terminal device, based on the address of the first terminal device carried in the video data, the port corresponding to the video capture and encoding module in the first terminal device (corresponding to the first port described above), and the content recorded by the conference server in step 4, the conference server can determine that the video data belongs to the first user. Therefore, it needs to send the video data to other users participating in the same conference as the first user (including the second user). As introduced above, the conference server records the correspondence between the first conference, the second user, and the second media parameters. Based on this, the conference server can know that the second user is also a participant in the first conference, and that the second user is different from the first user. Then, the conference server forwards the aforementioned video data to the second user according to the second media parameters: based on the address of the second terminal device in the second media parameters + the port corresponding to the video decoding and rendering module in the second terminal device (as the destination IP and destination port), the conference server sends the aforementioned video data to terminal device X so that the second user can view the video data on terminal device X.

[0173] Step 10. Terminal device X performs video decoding and rendering.

[0174] Specifically, when terminal device X receives the aforementioned video data sent by the conference server, it decodes the video data through the video decoding and rendering module in the conference application on terminal device X, and then renders and displays the decoded video data, so that the second user can view the video data from the first user on terminal device X.

[0175] Step 11. The first terminal device or the second terminal device detects the first user's first transfer operation.

[0176] The first transfer operation instruction transfers the function of the video acquisition and encoding module (corresponding to the first module described above) of the conferencing application in the first terminal device to the second terminal device. That is, the video data of the first user is no longer acquired and encoded from the first terminal device, but is acquired and encoded from the second terminal device.

[0177] Regarding the specific method of the first transfer operation, please refer to Method 1 and Method 2 described in step S203. Method 1 involves the first user operating on the first terminal device, which then generates a first instruction. This first instruction instructs the second terminal device to replace the video capture and encoding module in the first terminal device to process the first data. The first data refers to the conference data related to the function of the video capture and encoding module in the first terminal device during the first conference. Method 2 involves the first user operating on the second terminal device, which then generates a first instruction and subsequently sends it to the first terminal device.

[0178] Step 12. The first terminal device sends the video acquisition and encoding module to the second terminal device.

[0179] Specifically, assuming that the second terminal device does not have a video capture and encoding module before step 12, it cannot provide the functionality of this module to the first user on the second terminal device. Then, in response to the first transfer operation in step 11, the first terminal device can copy the video capture and encoding module on its own device to the second terminal device (i.e., step 12). The copying here essentially means installing a module with the same functionality as the video capture and encoding module in the first terminal device on the second terminal device. Subsequently, the second terminal device can run the video capture and encoding module on its own device, allowing the first user to transfer from the first terminal device to the second terminal device to enjoy the functionality of the video capture and encoding module during the first meeting.

[0180] If the second terminal device already contains a video capture and encoding module before step 12, then the first terminal device does not need to copy the video capture and encoding module to the second terminal device, and step 12 can be omitted.

[0181] Step 13. The second terminal device sends its address and second port to the first terminal device.

[0182] The second port is the port corresponding to the video acquisition and encoding module (corresponding to the second module described above) on the second terminal device.

[0183] Step 14. The first terminal device sends the address and second port of the second terminal device to the conference server.

[0184] Specifically, when the first terminal device obtains the address of the second terminal device and the second port corresponding to the video capture and encoding module in the second terminal device, the first terminal device sends parameter update information carrying the IP address and second port of the second terminal device to the conference server (refer to the description in step S204) to renegotiate how to transmit the video data of the first conference between the first user and the conference server. At this time, the parameter update information indicates that the first user expects to use the IP address and second port of the second terminal device to transmit video data with the conference server.

[0185] Step 15. The conference server sends a confirmation message to the first terminal device.

[0186] Specifically, when the conference server receives the address and second port of the second terminal device sent by the first terminal device, the conference server changes the previously recorded first correspondence between the first user and the address and first port of the first terminal device to a second correspondence between the first user and the address and second port of the second terminal device. Then, the conference server sends an acknowledgment message to the first terminal, indicating that the conference server confirms the transmission of video data of the first conference between the first user and the conference server using the address and second port of the second terminal device.

[0187] Step 16. The first terminal device instructs the second terminal device to run the video capture and encoding module.

[0188] Specifically, when the first terminal device receives the confirmation information sent by the conference server, it sends an instruction to the second terminal device (corresponding to the second instruction in step S205) to instruct the second terminal device to run the video capture and encoding module. The first terminal device can also send the address and port of the conference server, as well as parameters related to the video capture and encoding module, such as the encoding / decoding protocol type, resolution, and bandwidth (consistent with those in the first media parameters), to the second terminal device, allowing the first user to enjoy the module's functionality on the second terminal device and participate in the first conference. Alternatively, the conference server can send the encoding / decoding protocol type, resolution, and bandwidth parameters to the second terminal device before step 16.

[0189] Step 17. The second terminal device performs video acquisition and encoding.

[0190] Step 18. The second terminal device sends video data to the conference server.

[0191] Specifically, when the second terminal device receives the instruction from the first terminal device in step 16, the second terminal device starts running the video capture and encoding module. This allows the first user to enjoy the functions of the video capture and encoding module while participating in the first meeting. Specifically, the second terminal device captures video data through its camera and encodes the video data using its processor. Then, the second terminal device sends the encoded video data to the meeting server, which forwards it to the devices of other users participating in the same meeting as the first user (including the second user's terminal device X).

[0192] Step 19. The conference server sends video data to terminal device X.

[0193] Specifically, when the conference server receives video data (encoded) sent by the second terminal device, based on the address of the second terminal device carried in the video data, the port corresponding to the video acquisition and encoding module in the second terminal device (corresponding to the second port described above), and the information changed by the conference server in step 12, the conference server can determine that the video data belongs to the first user, and then needs to send the video data to other users (including the second user) participating in the same conference as the first user.

[0194] Step 20. Terminal device X performs video decoding and rendering.

[0195] Specifically, terminal device X runs a conferencing application (including a video decoding and rendering module) based on its hardware. When terminal device X receives video data (encoded) sent by the conferencing server, it decodes the video data and then renders and displays the video data on terminal device X.

[0196] It should be noted that, Figure 8 This application only describes the process of transferring the video acquisition and encoding module functionality from the first terminal device to the second terminal device. In reality, the first terminal device can also transfer other modules in the conferencing application to the second terminal device. Furthermore, the first terminal device can also transfer the functionality of modules in the conferencing application to other peer devices (not the second terminal device), but this application does not impose specific limitations on this.

[0197] Similarly, the second user can also transfer some modules of the conferencing application from terminal device X to the peer device of terminal device X, enabling the second user to participate in the first meeting together based on terminal device X and its peer device, thus improving the second user's meeting experience. For example, assuming that terminal device Y and terminal device X are both located in the second physical space where the second user is located, and terminal device Y is the peer device of terminal device X, and terminal device Y supports running a video decoding and rendering module, the second user can choose to transfer the video decoding and rendering module from terminal device X to terminal device Y. Subsequently, when the conferencing server receives the first user's video data, it will forward the first user's video data to the video decoding and rendering module of terminal device Y (instead of forwarding it to terminal device X), allowing the second user to view the first user's video data on terminal device Y.

[0198] In summary, in the meeting participation method provided in this application embodiment, the user can transfer the functionality of some modules of the meeting application in the terminal device currently used for meeting participation to the peer device. That is, the peer device replaces some modules in the terminal device to process the corresponding meeting data (i.e., all meeting data related to the functionality of those modules). Thus, the user can participate in the meeting using both the terminal device and its peer device (essentially treating these multiple devices as a super terminal with more powerful functionality than a single terminal device), enabling cross-device meeting participation and fully leveraging the advantages of multiple devices (fully utilizing the strengths of each device, with each device complementing the others), thus improving the user's meeting experience. Furthermore, since the functionality of some modules of the meeting application in the terminal device is transferred to the peer device, these modules will utilize the processor and other computing resources of the peer device, thereby reducing the processor load on the terminal device and also reducing its power consumption and heat generation.

[0199] Please see Figure 9 , Figure 9 This is a flowchart illustrating another meeting participation method provided in this application embodiment, including steps S901 to S905.

[0200] S901, In response to the first user's operation of entering the first conference on the first terminal device, the first terminal device sends the first media parameters to the conference server.

[0201] For details on step S901, please refer to the description of step S201, which will not be repeated here.

[0202] S902, The conference server records the correspondence between the first conference, the first user, and the first media parameters, and sends the confirmation result of the first media parameters to the first terminal device.

[0203] For details on step S902, please refer to the description of step S202, which will not be repeated here.

[0204] S903. The first terminal device obtains a first instruction, wherein the first instruction instructs the second terminal device to replace the first module in the conference application of the first terminal device to process the first data, and the first data is part of the conference data related to the function of the first module in the first conference.

[0205] For information on the conference application, the second terminal device, and the first module, please refer to the introduction in steps S201 to S203; they will not be repeated here.

[0206] It should be noted that, Figure 2 The first data in the embodiment is all meeting data related to the function of the first module in the first meeting, and correspondingly, Figure 2The first instruction indicates that the second terminal device will replace the first module in the first terminal device to process all the above-mentioned conference data. In other words, the function of the first module will be completely transferred to the second terminal device. The second terminal device (the second module) completely replaces the first module to perform the corresponding functions. The first module will no longer provide the corresponding functions to the first user. Therefore, the first terminal device can stop running the first module after the above transfer is completed to save energy.

[0207] And in Figure 9 In this embodiment, the first data is a portion of the meeting data related to the function of the first module in the first meeting. Accordingly, Figure 9 The first instruction in the embodiment indicates that the second terminal device should replace the first module to process a portion of the conference data. The other portion of data related to the function of the first module can be processed by the first module, or it can be processed by other peer devices of the first terminal device (excluding the second terminal device). That is to say, Figure 9 The second terminal device in this embodiment does not completely replace the first module in the first terminal device. Instead, it processes a portion of all the conference data that the first module needs to process on the second terminal device, which can reduce the processor load of the first terminal device to a certain extent.

[0208] This application does not specify the method for obtaining the first instruction; examples of method one and method two will be provided below.

[0209] Method 1: In response to the first user's first operation on the first terminal device, the first terminal device generates a first instruction.

[0210] In this approach, the first terminal device automatically generates a first instruction based on a first operation performed by a first user on the first terminal device. The first operation includes instructing the operation of the first module, instructing the operation of the second terminal device, and instructing the aforementioned first data. This application does not specifically limit these operations.

[0211] For example, suppose the first terminal device is a tablet, and the first module in the conferencing application on the tablet is a video decoding and rendering module, which is used for video decoding and rendering. Figure 10 As shown, when the first terminal device receives encoded video data from multiple users (all users of the first conference but different from the first user) sent by the server, the video decoding and rendering module can display the video data of these multiple users in separate windows on the device's screen, with each user's video data displayed in a separate video window. Alternatively, the video data captured by the first terminal device's camera can also be displayed on the screen, i.e., the first user's video data can be displayed in the first user's video window.

[0212] Suppose the first user wants to transfer the second user's video data (corresponding to the first data) to their mobile phone for display. The first user can perform the following operation on their mobile phone: select the second user's video window on the first terminal device and shake the tablet close to the phone (i.e., a screen-shaking operation). Based on this operation, the first terminal device identifies the video encoding and rendering module in the tablet as the first module and the mobile phone as the second terminal device, thus generating a first instruction. This first instruction instructs the mobile phone to replace the video decoding and rendering module in the tablet to process the second user's video data, so that the second user's video data can be displayed on the mobile phone. Then, the first terminal device sends the first instruction to the second terminal device. The first terminal device also forwards the second user's video data from the conference server to the second terminal device. The first terminal device no longer processes the second user's video data, and the second user's video window is no longer displayed on the first terminal device. The video data of other users is still processed by the video capture and encoding module in the first terminal device, and the video windows of the other users will still be displayed on the screen of the first terminal device.

[0213] It should be noted that, Figure 9 The user operation methods, video window sizes, positions, and numbers shown are merely examples and do not constitute limitations. In real-world applications, more or fewer users may participate in the first meeting, resulting in more or fewer users' video data to be displayed, and consequently, more or fewer video windows. Figure 9 After the second user's video window on the tablet is transferred to the mobile phone for display, the size of the video windows of the other users on the tablet may remain unchanged or the window layout may be automatically adjusted; this application does not impose specific limitations on this. The first user may also choose to transfer the video data of other users besides the second user to the peer device of the first terminal device for display, and the peer device will display the corresponding video windows of the other users' video data; this application does not impose specific limitations on this.

[0214] Method 2: In response to the second operation of the first user on the second terminal device, the second terminal device generates a first instruction, and the first terminal device receives the first instruction sent by the second terminal device.

[0215] Unlike Method 1, where the first terminal device actively generates the first instruction, in Method 2, the first terminal device does not actively generate the first instruction, but instead receives the first instruction from the second terminal device. The first instruction is generated by the second terminal device based on the second operation of the first user on the second terminal device. The second operation includes operations that instruct the first module and operations that instruct the aforementioned first data. This application does not specifically limit these operations.

[0216] For example, Figure 11An example menu displayed on a second terminal device is provided, including options such as a chat window, a sharing window, and a video window. The meeting collaboration module in the meeting application processes meeting collaboration information such as shared content and chat content, thereby displaying the sharing window corresponding to the shared content and the chat window corresponding to the chat content. The video decoding and rendering module in the meeting application processes video data, thereby displaying the video window corresponding to the video data. Suppose a first user wants to transfer the sharing window displayed on the first terminal device to the second terminal device. The first user can select the sharing window option in the menu on the second terminal device. The second terminal device generates a first instruction based on the user's operation. This first instruction instructs the second terminal device to replace the meeting collaboration module (the first module at this time) in the first terminal device to process the shared content (the first data at this time), and then the second terminal device sends the first instruction to the first terminal device.

[0217] It should be noted that, Figure 11 The shared window will be transferred from the first terminal device to the second terminal device for display. This means that the second terminal device replaces the meeting collaboration module in the first terminal device to implement some functions, but other functions of the meeting collaboration module (including displaying chat windows, annotation windows, etc.) are still implemented on the first terminal device. Figure 11 The user operation methods, the number and content of options displayed in the menu are merely examples and do not constitute limitations. In practical application scenarios, the first user can also perform other operations on the second terminal device to instruct the first module and the first data, and this application does not impose specific limitations.

[0218] S904. The first terminal device sends a second instruction to the second terminal device according to the first instruction, wherein the second instruction instructs the second terminal device to run a second module in the second terminal device to process the first data. The second module has the same function as the first module. The first data processed by the second terminal device is either sent to the second terminal device by the first terminal device / conference server or collected by the second terminal device.

[0219] In order to implement the same functions as the first module on the second terminal device (to fulfill the requirements of the first instruction), Figure 9 The second terminal device in this embodiment also needs to install a module with the same function as the first module, namely the second module mentioned above. The second module can be one that is already installed on the second terminal device, or it can be obtained by copying the first module to the second terminal device from the first terminal device, or it can be obtained by the first terminal device instructing the second terminal device to actively download and install it from an application store or application website after receiving the first instruction. For details, please refer to the relevant description in step S204, which will not be repeated here.

[0220] It should be understood that when the second terminal device runs the second module to replace the first module in the first terminal device to perform some functions, the second module may need to transmit data with the conference server. The data transmission direction can be uplink, that is, from the second terminal device to the conference server to transmit the first data processed by the second module, or downlink, that is, from the conference server to the second terminal device to transmit the first data that the second module needs to process.

[0221] The data transmission direction described above depends on the type of the first data. For example, suppose the first data is shared content from this device, and the second module is a conference collaboration module. This module needs to collect the shared content from this device and send the encoded content to the conference server so that the conference server can forward the shared content to other users' devices for display. In this case, the data transmission direction is upstream. As another example, suppose the first data is shared content from other users' devices, and the second module is a conference collaboration module. This module needs to decode and render the shared content from other users' devices sent by the conference server, and then display the shared content in the shared window on this device's screen. In this case, the data transmission direction is downstream. Yet another example, suppose the first data is video data from a second user participating in the same conference as the first user, and the second module is a video decoding and rendering module. This module needs to decode the second user's video data sent by the conference server, and then display the second user's video data on this device's screen. In this case, the data transmission direction is downstream.

[0222] The second terminal device can transmit data with the conference server by forwarding data from the first terminal device.

[0223] Optionally, if the first data processed by the second terminal device is data collected by the second terminal device (i.e., the data transmission direction is uplink), the second terminal device can send the processed first data to the first terminal device. Correspondingly, the first terminal device receives the processed first data sent by the second terminal device and then sends the processed first data to the conference server.

[0224] Optionally, if the first data processed by the second terminal device was sent by the first terminal device (i.e., the data transmission direction is downlink), the first terminal device can receive the first data sent by the conference server and then send the first data to the second terminal device. The second terminal device receives the first data sent by the first terminal device and then processes the first data through its second module.

[0225] S905, The second terminal device runs the second module to process the first data according to the second instruction.

[0226] It should be noted that, Figure 9This describes using a second module in a second terminal device to replace the first module in a first terminal device to process the first data. The first data is only a portion of the conference data related to the functions of the first module. Therefore, the second module in the second terminal device only replaces the first module in the first terminal device to implement some functions, rather than completely replacing the first module to implement all functions. In fact, the first user can also refer to... Figure 9 The method involves transferring some functions of other modules in the conferencing application of the first terminal device, excluding the first module, to the second terminal device or other peer devices. For the sake of brevity, this will not be elaborated here.

[0227] In summary, in the meeting participation method provided in this application embodiment, users can transfer some functions of some modules of the meeting application in the terminal device currently used for meeting participation to the peer device. That is, the peer device replaces some modules in the terminal device to process some meeting data (i.e., the meeting data related to the functions of those modules). Thus, the user can participate in the meeting using both the terminal device and its peer device (essentially treating these multiple devices as a super terminal with more powerful functions than a single terminal device), enabling cross-device meeting participation and fully leveraging the advantages of multiple devices (fully utilizing the strengths of each device, with each device complementing the others), thus improving the user's meeting experience. Furthermore, because some functions of some modules of the meeting application in the terminal device are transferred to the peer device, the processor load on the terminal device can be reduced, as can its power consumption and heat generation.

[0228] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a meeting participation device 1200 provided in an embodiment of this application. The meeting participation device 1200 can be deployed in a first terminal device. The meeting participation device 1200 includes an acquisition unit 1201 and a sending unit 1202.

[0229] The acquisition unit 1201 is used to acquire a first instruction, wherein the first user enters the first conference based on the conference application in the first terminal device. The conference application includes multiple modules with different functions. The first instruction instructs the use of a second terminal device to replace the first module among the multiple modules to process the first data. The first data is part or all of the conference data in the first conference that is related to the function of the first module.

[0230] The sending unit 1202 is used by the first terminal device to send a second instruction to the second terminal device according to the first instruction. The second instruction instructs the second terminal device to run a second module in the second terminal device to process the first data. The second module has the same function as the first module. The first data processed by the second terminal device is sent to the second terminal device by the first terminal device / conference server or collected by the second terminal device.

[0231] Optionally, the acquisition unit 1201 is specifically configured to: in response to a first operation by a first user on a first terminal device, the first terminal device generates a first instruction, the first operation including instructing an operation of a first module and instructing an operation of a second terminal device; or, the first terminal device receives a first instruction from a second terminal device, the first instruction being generated by the second terminal device based on a second operation by the first user on the second terminal device, the second operation including instructing an operation of a first module.

[0232] Optionally, the above-mentioned operation of the first module includes one or more of the following: selecting the hardware required for the first module to run, selecting the identifier of the first module, and performing the corresponding motion-sensing operation of the first module.

[0233] Optionally, the aforementioned hardware includes one or more of the following: microphone, speaker, camera, screen, processor, and sensor.

[0234] Optionally, after the acquisition unit 1201 acquires the first instruction, the acquisition unit 1201 is further configured to: acquire the IP address of the second terminal device and the port corresponding to the second module in the second terminal device, provided that the first data is all conference data related to the function of the first module in the first conference. The sending unit 1202 is further configured to: send parameter update information to the conference server, wherein the parameter update information includes the IP address of the second terminal device and the port corresponding to the second module in the second terminal device. The acquisition unit 1201 is further configured to: receive confirmation information sent by the conference server, wherein the confirmation information instructs the conference server to change the recorded first correspondence to a second correspondence, wherein the first correspondence is the correspondence between the IP addresses of the first user and the first terminal device and the port corresponding to the first module in the first terminal device, and the second correspondence is the correspondence between the IP addresses of the first user and the second terminal device and the port corresponding to the second module in the second terminal device.

[0235] Optionally, the conference participation device 1200 also includes a control unit 1203: after the acquisition unit 1201 receives the confirmation information sent by the conference server, the control unit 1203 is used to stop the operation of the first module.

[0236] Optionally, after the acquisition unit 1201 receives the confirmation information sent by the conference server, the sending unit 1202 is further configured to: send parameters related to the first module to the second terminal device, wherein the parameters related to the first module include one or more of the following: encoding / decoding protocol type, resolution, bandwidth, transmission protocol type, IP address of the conference server, and port.

[0237] Optionally, after the sending unit 1202 sends the second instruction to the second terminal device according to the first instruction, the acquiring unit 1201 is further configured to: receive the processed first data sent by the second terminal device if the first data processed by the second terminal device is collected by the second terminal device. The sending unit 1202 is further configured to: send the aforementioned processed first data to the conference server.

[0238] Optionally, after the sending unit 1202 sends the second instruction to the second terminal device according to the first instruction, the obtaining unit 1201 is further configured to: receive the first data sent by the conference server if the first data processed by the second terminal device was sent by the first terminal device. The sending unit 1202 is further configured to: send the first data to the second terminal device.

[0239] Optionally, after the acquisition unit 1201 acquires the first instruction, the sending unit 1202 is further configured to: copy the first module to the second terminal device to obtain the second module in the second terminal device.

[0240] Optionally, before the acquisition unit 1201 acquires the first instruction, the acquisition unit 1201 is further configured to: receive device information sent by the second terminal device, the device information indicating that the second terminal device supports running the first module.

[0241] Optionally, the first module may include one or more of the following modules: conference control module, conference collaboration module, artificial intelligence processing module, audio acquisition and encoding module, audio decoding and playback module, video acquisition and encoding module, and video decoding and rendering module.

[0242] It should be noted that, Figure 12 The meeting participation device 1200 is exemplarily divided into an acquisition unit 1201, a transmission unit 1202, and a control unit 1203 based solely on function. In practice, the meeting participation device 1200 may also include more or fewer units. For example, a single unit may be split into multiple units, two or more units may be merged into one unit, and other units may be added to the meeting participation device 1200. This application does not limit this. The acquisition unit 1201, transmission unit 1202, and control unit 1203 can all be implemented in software or in hardware.

[0243] Please see Figure 13, Figure 13 This is a schematic diagram of another meeting participation device 1300 provided in this application embodiment. The meeting participation device 1300 can be deployed in a second terminal device. The meeting participation device 1300 includes a receiving unit 1301 and a control unit 1302.

[0244] The receiving unit 1301 is configured to: receive a second instruction sent by the first terminal device according to the first instruction, wherein the first user enters the first conference according to the conference application in the first terminal device, the conference application includes multiple modules with different functions, the first instruction instructs the second terminal device to replace the first module among the multiple modules to process the first data, and the first data is part or all of the conference data in the first conference related to the function of the first module.

[0245] The control unit 1302 is used to: run the second module in the second terminal device to process the first data according to the second instruction, wherein the second module has the same function as the first module, and the first data processed by the second terminal device is sent to the second terminal device by the first terminal device / conference server or collected by the second terminal device.

[0246] Optionally, the conference participation device 1300 further includes a sending unit 1303. Before the receiving unit 1301 receives the second instruction sent by the first terminal device according to the first instruction, the sending unit 1303 is configured to: generate the first instruction in response to a second operation by the first user on the second terminal device, the second operation including instructing the operation of the first module, and then send the first instruction to the first terminal device.

[0247] Optionally, the above-mentioned operation of the first module includes one or more of the following: selecting the hardware required for the first module to run, selecting the identifier of the first module, and performing the corresponding motion-sensing operation of the first module.

[0248] Optionally, the aforementioned hardware includes one or more of the following: microphone, speaker, camera, screen, processor, and sensor.

[0249] Optionally, the receiving unit 1301 is further configured to: receive parameters related to the first module sent by the first terminal device when the first data is all conference data related to the function of the first module in the first conference. The control unit 1302 is further configured to: process the first data according to the parameters related to the first module and the second module, wherein the parameters related to the first module include one or more of the following: encoding / decoding protocol type, resolution, bandwidth, transmission protocol type, IP address of the conference server, and port.

[0250] Optionally, the sending unit 1303 is further configured to: send the first data processed by the second module to the first terminal device when the first data is part of the conference data related to the function of the first module in the first conference and the first data processed by the second module is collected by the second terminal device.

[0251] Optionally, the receiving unit 1301 is further configured to: receive the first data sent by the first terminal device when the first data is part of the conference data related to the function of the first module in the first conference and the first data processed by the second module is sent by the first terminal device.

[0252] Optionally, the second module is obtained by copying the first module to the second terminal device from the first terminal device.

[0253] Optionally, before the receiving unit 1301 receives the second instruction sent by the first terminal device according to the first instruction, the sending unit 1303 is further configured to: send device information to the first terminal device, wherein the device information indicates that the second terminal device supports running a module with the same function as the first module.

[0254] Optionally, the first module may include one or more of the following modules: conference control module, conference collaboration module, artificial intelligence processing module, audio acquisition and encoding module, audio decoding and playback module, video acquisition and encoding module, and video decoding and rendering module.

[0255] It should be noted that, Figure 13 The conference participation device 1300 is exemplarily divided into a receiving unit 1301, a control unit 1302, and a transmitting unit 1303 based solely on function. In practice, the conference participation device 1300 may also contain more or fewer units. For example, a single unit may be split into multiple units, two or more units may be merged into one unit, and other units may be added to the conference participation device 1300. This application does not limit this. The receiving unit 1301, control unit 1302, and transmitting unit 1303 can all be implemented in software or in hardware.

[0256] Please see Figure 14 This application also provides a terminal device 1400, including a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402. The terminal device 1400 can be a laptop, tablet, desktop computer, smartphone, wearable device, etc., and this application does not specifically limit it, nor does it limit the number of processors and memories in the terminal device 1400.

[0257] Bus 1402 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus 1402 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1402 may include a path for transmitting information between various components of the terminal device 1400 (e.g., memory 1406, processor 1404, communication interface 1408).

[0258] The processor 1404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0259] The memory 1406 may include volatile memory, such as random access memory (RAM). The processor 1404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0260] The memory 1406 stores executable program code. The processor 1404 executes this executable program code to implement... Figure 12 The functions of the acquisition unit 1201, the transmission unit 1202, and the control unit 1203 are realized to achieve the following: Figure 2 or Figure 9 The steps involved in the method by the first terminal device. Alternatively, the processor 1404 executes the executable program code to implement respectively. Figure 14 The functions of the receiving unit 1301, the control unit 1302, and the transmitting unit 1303 are realized to achieve the following: Figure 2 or Figure 9 The steps involved in the second terminal device in the method.

[0261] The communication interface 1408 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the terminal device 1400 and other devices or communication networks.

[0262] As one possible implementation, the terminal device 1400 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit provides power to the processor, and the processor performs execution... Figure 2 or Figure 8 The steps involved in the method involving the first or second terminal device are omitted here for brevity. The processor can be implemented using a CPU, or it can be implemented using computing devices or AI chips such as GPUs, DPUs, NPUs, XPUs, SoCs, offloading cards, or accelerator cards.

[0263] As one possible implementation, the terminal device 1400 may include multiple types of processors 1404, meaning the terminal device 1400 is a heterogeneous device. For example, the terminal device 1400 may include a CPU and a GPU, and execution can be performed by at least one of the processors 1404. Figure 2 or Figure 9 The steps involved in the method involving the first or second terminal device are omitted here for brevity.

[0264] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a terminal device can store. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that, when executed by a terminal device, cause the terminal device to perform actions such as... Figure 2 or Figure 9 The steps involved in the method involving the first terminal device or the second terminal device.

[0265] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a terminal device or stored on any usable medium. When the computer program product is run by the terminal device, it causes the terminal device to perform actions such as... Figure 2 or Figure 9 The steps involved in the method involving the first terminal device or the second terminal device.

[0266] Finally, it should be noted that the embodiments described above are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. The terminology of this application is for the purpose of describing specific embodiments only and is not intended to limit this application. The singular forms "a," "described," and "the" of this application are also intended to include the plural forms, unless the context clearly indicates otherwise.

Claims

1. A method for participating in a meeting, characterized in that, The method is applied to a first terminal device, and the method includes: The first terminal device receives a first instruction, wherein the first user enters the first meeting based on a meeting application in the first terminal device, the meeting application includes multiple modules with different functions, the first instruction instructs the second terminal device to replace the first module among the multiple modules to process the first data, the first data being part or all of the meeting data in the first meeting related to the function of the first module; The first terminal device sends a second instruction to the second terminal device according to the first instruction. The second instruction instructs the second terminal device to run a second module in the second terminal device to process the first data. The second module has the same function as the first module. The first data processed by the second terminal device is either sent to the second terminal device by the first terminal device / conference server or collected by the second terminal device.

2. The method according to claim 1, characterized in that, The first terminal device acquires a first instruction, including: In response to a first operation by the first user on the first terminal device, the first terminal device generates the first instruction, wherein the first operation includes instructing the operation of the first module and instructing the operation of the second terminal device; Alternatively, the first terminal device receives the first instruction from the second terminal device, wherein the first instruction is generated by the second terminal device based on a second operation performed by the first user on the second terminal device, the second operation including an instruction to the first module.

3. The method according to claim 2, characterized in that, The operation of instructing the first module includes one or more of the following: selecting the hardware required for the first module to run, selecting the identifier of the first module, and performing the somatosensory operation corresponding to the first module.

4. The method according to claim 3, characterized in that, The hardware includes one or more of the following: microphone, speaker, camera, screen, processor, and sensor.

5. The method according to any one of claims 1 to 4, characterized in that, After the first terminal device obtains the first instruction, the method further includes: In the case that the first data is all the meeting data related to the function of the first module in the first meeting, the first terminal device obtains the Internet Protocol IP address of the second terminal device and the port corresponding to the second module in the second terminal device; The first terminal device sends parameter update information to the conference server, wherein the parameter update information includes the IP address of the second terminal device and the port corresponding to the second module in the second terminal device; The first terminal device receives confirmation information sent by the conference server, wherein the confirmation information instructs the conference server to change the recorded first correspondence to a second correspondence. The first correspondence is the correspondence between the IP addresses of the first user and the first terminal device and the port corresponding to the first module in the first terminal device, and the second correspondence is the correspondence between the IP addresses of the first user and the second terminal device and the port corresponding to the second module in the second terminal device.

6. The method according to claim 5, characterized in that, After the first terminal device receives the confirmation information sent by the conference server, the method further includes: The first terminal device stops running the first module.

7. The method according to claim 5 or 6, characterized in that, After the first terminal device receives the confirmation information sent by the conference server, the method further includes: The first terminal device sends parameters related to the first module to the second terminal device. The parameters related to the first module include one or more of the following: encoding / decoding protocol type, resolution, bandwidth, transmission protocol type, IP address of the conference server, and port.

8. The method according to any one of claims 1 to 4, characterized in that, After the first terminal device sends a second instruction to the second terminal device according to the first instruction, the method further includes: When the first data processed by the second terminal device is collected by the second terminal device, the first terminal device receives the processed first data sent by the second terminal device and sends the processed first data to the conference server; If the first data processed by the second terminal device was sent by the first terminal device, the first terminal device receives the first data sent by the conference server and sends the first data to the second terminal device.

9. The method according to any one of claims 1 to 8, characterized in that, After the first terminal device obtains the first instruction, the method further includes: The first terminal device copies the first module to the second terminal device, thus obtaining the second module in the second terminal device.

10. The method according to any one of claims 1 to 9, characterized in that, Before the first terminal device obtains the first instruction, the method further includes: The first terminal device receives device information sent by the second terminal device, wherein the device information indicates that the second terminal device supports running the first module.

11. The method according to any one of claims 1 to 10, characterized in that, The first module includes one or more of the following: a conference control module, a conference collaboration module, an artificial intelligence processing module, an audio acquisition and encoding module, an audio decoding and playback module, a video acquisition and encoding module, and a video decoding and rendering module.

12. A method for participating in a meeting, characterized in that, The method is applied to a second terminal device, and the method includes: The second terminal device receives a second instruction sent by the first terminal device according to the first instruction, wherein the first user enters the first meeting according to the meeting application in the first terminal device, the meeting application includes multiple modules with different functions, the first instruction instructs the second terminal device to replace the first module among the multiple modules to process the first data, the first data being part or all of the meeting data in the first meeting related to the function of the first module; The second terminal device runs the second module in the second terminal device according to the second instruction to process the first data, wherein the second module has the same function as the first module, and the first data processed by the second terminal device is sent to the second terminal device by the first terminal device / conference server or collected by the second terminal device.

13. A meeting participation device, characterized in that, Includes units for performing the method as described in any one of claims 1-11 or 12.

14. A terminal device, characterized in that, It includes a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the terminal device to perform the method as described in any one of claims 1-11 or 12.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the terminal device, the terminal device performs the method as described in any one of claims 1-11 or 12.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions that, when executed by a terminal device, cause the terminal device to perform the method as described in any one of claims 1-11 or 12.

Images