Method and device for transmitting data in video conference

Abstract

The embodiment of the invention provides a method and device for transmitting data in a video conference, and the method comprises the steps: generating a subscription message which comprises an identifier of a producer subscribed by a user terminal and resolution indication information of a subscribed video stream, the resolution indication information is used for indicating the resolution of a video stream subscribed by the user terminal, and the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode; the subscription message is sent to a signaling server, so that the signaling server generates a subscription indication message according to the subscription message, and the subscription indication message is used for indicating a media forwarding server to send a video stream with a corresponding resolution to the user terminal; and receiving the video stream sent by the media forwarding server. According to the embodiment of the invention, the video streams with different resolutions can be loaded for the user terminal according to the display view mode, and the problem of resource waste in a video conference is reduced.

Description

Technical Field

[0001] The present application relates to the field of video conferencing, and in particular, to a method and device for transmitting data in a video conferencing. Background Art

[0002] With the development of communication technology, video conferencing has become an increasingly important means of communication in various communication scenarios, such as work communication, life communication, team communication, campus communication, etc.

[0003] In order to display diversity and meet different needs, video conferencing has multiple display view modes, such as main view mode and gallery mode, and different modes or pages in the same mode can be switched. However, existing video conferencing wastes resources when switching screens.

[0004] Therefore, how to reduce the waste of resources in video conferencing has become an urgent problem to be solved. Summary of the invention

[0005] An object of an embodiment of the present application is to provide a method and device for transmitting data in a video conference. The technical solution of the embodiment of the present application can reduce the problem of resource waste in a video conference.

[0006] In a first aspect, an embodiment of the present application provides a method for transmitting data in a video conference, comprising: generating a subscription message, the subscription message including an identifier of a producer subscribed by a user terminal, and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate the resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the preset plurality of resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; sending the subscription message to a signaling server so that the signaling server generates a subscription indication message based on the subscription message, the subscription indication message being used to instruct a media forwarding server to send a video stream of corresponding resolution to the user terminal; and receiving the video stream sent by the media forwarding server.

[0007] In the embodiment of the present application, consumers send corresponding subscription messages to the signaling server according to different needs, so that consumers can flexibly load video streams with different resolutions according to different display view modes, thereby reducing bandwidth usage. Compared with the prior art in which all video streams are loaded with fixed resolutions, resource overhead can be saved and resource waste can be reduced.

[0008] In one embodiment, generating the subscription message includes: generating the subscription message according to the user's operation instructions, the user's operation instructions include entering a video conference operation instruction or adjusting a video screen operation instruction; wherein the video screen adjustment operation instruction includes: sliding operation instructions, clicking operation instructions, switching view operation instructions, gesture operation instructions, shaking the terminal operation instructions or voice control operation instructions.

[0009] In one embodiment, the current mode is the main view mode, and the user's operation instruction is used to switch the current model from the main view mode to the gallery mode. The resolution of the video stream subscribed by the user terminal is the second resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0010] In one implementation, the current mode is a gallery mode, the user's operation instruction is used to turn pages in the gallery mode, and the resolution of the video stream subscribed by the user terminal is the second resolution.

[0011] In one embodiment, the current mode is the gallery mode, and the operation instruction is used to switch the current model to the main view mode. The resolution of the video stream subscribed by the user terminal is the first resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0012] In one embodiment, after the action of adjusting the video screen corresponding to the user's operation instruction is completed, the method also includes: sending a cancellation subscription message to the signaling server, so that the signaling server generates a cancellation indication message according to the cancellation subscription message, and the indication message is used to instruct the media forwarding server to cancel sending the video stream before the screen is adjusted to the user terminal, wherein the cancellation subscription message includes an identifier of the video stream before the screen is adjusted.

[0013] In one embodiment, the first resolution includes a first sub-resolution or a second sub-resolution, wherein the first sub-resolution is greater than the second sub-resolution, and both the first sub-resolution and the second sub-resolution are greater than the second resolution, and the network status of the user terminal corresponding to the first sub-resolution is better than the network status of the user terminal corresponding to the second sub-resolution.

[0014] In a second aspect, an embodiment of the present application provides a method for transmitting data in a video conference, comprising: receiving a subscription message sent by a user terminal, the subscription message including an identifier of a producer subscribed by the user terminal, and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate the resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the preset plurality of resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; generating a subscription indication message according to the subscription message, the subscription indication message being used to instruct a media forwarding server to send a video stream of corresponding resolution to the user terminal; and sending the subscription indication message to a media forwarding server.

[0015] In a third aspect, an embodiment of the present application provides a device for transmitting data in a video conference, comprising: a generating unit, configured to generate a subscription message, the subscription message comprising an identifier of a producer subscribed by a user terminal, and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate the resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the preset plurality of resolutions comprising a first resolution and a second resolution, the first resolution being greater than the second resolution; a sending unit, configured to send the subscription message to a signaling server, so that the signaling server generates a subscription indication message according to the subscription message, the subscription indication message being used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal; and a receiving unit, configured to receive the video stream sent by the media forwarding server.

[0016] In one embodiment, the generating unit is specifically used to generate the subscription message according to the user's operation instruction, and the user's operation instruction includes an operation instruction for entering a video conference or an operation instruction for adjusting a video screen. The operation instruction for adjusting a video screen includes: a sliding operation instruction, a clicking operation instruction, a view switching operation instruction, a gesture operation instruction, a shaking terminal operation instruction, or a voice control operation instruction.

[0017] In one embodiment, the current mode is the main view mode, and the user's operation instruction is used to switch the current model from the main view mode to the gallery mode. The resolution of the video stream subscribed by the user terminal is the second resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0018] In one implementation, the current mode is a gallery mode, the user's operation instruction is used to turn pages in the gallery mode, and the resolution of the video stream subscribed by the user terminal is the second resolution.

[0019] In one embodiment, the current mode is the gallery mode, and the operation instruction is used to switch the current model to the main view mode. The resolution of the video stream subscribed by the user terminal is the first resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0020] In one embodiment, after the action of adjusting the video screen corresponding to the user's operation instruction is completed, the sending unit is also used to send a cancellation subscription message to the signaling server, so that the signaling server generates a cancellation indication message according to the cancellation subscription message, and the indication message is used to instruct the media forwarding server to cancel sending the video stream before the screen is adjusted to the user terminal, wherein the cancellation subscription message includes an identifier of the video stream before the screen is adjusted.

[0021] In one embodiment, the first resolution includes a first sub-resolution or a second sub-resolution, wherein the first sub-resolution is greater than the second sub-resolution, and both the first sub-resolution and the second sub-resolution are greater than the second resolution, and the network status of the user terminal corresponding to the first sub-resolution is better than the network status of the user terminal corresponding to the second sub-resolution.

[0022] In a fourth aspect, an embodiment of the present application provides a device for transmitting data in a video conference, comprising: a receiving unit, configured to receive a subscription message sent by a user terminal, the subscription message including an identifier of a producer subscribed by the user terminal, and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate the resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the preset plurality of resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; a generating unit, configured to generate a subscription indication message according to the subscription message, the subscription indication message being used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal; and a sending unit, configured to send the subscription indication message to a media forwarding server.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in the first aspect and any embodiment of the first aspect or the method described in the second aspect, that is, any embodiment of the second aspect.

[0024] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it can implement the method described in the first aspect and any embodiment of the first aspect or the method described in the second aspect, i.e. any embodiment of the second aspect.

[0025] In the seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, it can implement the method described in the first aspect and any embodiment of the first aspect or the method described in the second aspect, that is, any embodiment of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solution of an embodiment of the present application, the drawings required for use in an embodiment of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a video conferencing system scenario provided for an embodiment of the present application;

[0028] Figure 2 A flow chart of a method for transmitting data in a video conference provided for one embodiment of the present application;

[0029] Figure 3 A schematic diagram of a device for transmitting data in a video conference provided for one embodiment of the present application;

[0030] Figure 4 A schematic diagram of a device for transmitting data in a video conference provided in accordance with another embodiment of the present application;

[0031] Figure 5 A schematic diagram of an electronic device is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0033] In a video conference, the user terminal supports multiple display view modes, such as main view mode and gallery mode, and different modes or pages in the same mode can be switched.

[0034] Gallery mode is a common way to display the layout of the user interface (UI) of participants in video conferencing. In gallery mode, the participants of the meeting are arranged on the screen in equal-sized squares. This grid is usually divided into four, six or nine layouts, depending on the settings of the conference platform, the settings of the user terminal and the number of participants. Each square displays the video of a participant. This layout makes it easier for participants to see the expressions and actions of all other people, promoting more natural and intimate communication.

[0035] The main view mode is a view mode in which a video conference displays a single user's screen, for example, the screen of the host or the main speaker. Regardless of the main view or gallery mode, if a user wants to view the screens of other users outside the view, they need to turn pages or switch views. In existing video conferences, there is a problem of wasting resources when switching screens.

[0036] For example, in the existing main view mode, the gallery view will appear when the user swipes left on the interface. To ensure that the view is displayed when sliding, it is necessary to load the next page of gallery view when sliding, such as loading 4 views of the next page, that is, when sliding, it is necessary to load the main view mode screen and the 4 video screens of the next page of gallery mode at the same time. When sliding between gallery pages, a maximum of 8 video screens need to be loaded at the same time. In the prior art, the resolution of the video screen loaded in both the main view mode and the gallery mode is fixed. Loading multiple videos at the same time will take up a lot of resources, resulting in a waste of resources. There is also a problem of high performance consumption of the terminal device, resulting in excessive device temperature, device freezes, and other problems.

[0037] In view of the above problems, the present application provides a method for transmitting data in a video conference, which can flexibly load video streams of different resolutions for the consumer's user terminal according to different display view modes. Since the sizes of a screen displayed in the main view mode and the gallery mode are different, in order to obtain the same viewing effect, the requirements for resolution are also different. The embodiment of the present application loads video streams of different resolutions in different modes, which can save resource overhead and reduce resource waste compared to the prior art that all video streams are loaded with fixed resolutions.

[0038] In addition, since it can reduce performance consumption, it can effectively reduce the problem of device overheating and lag, and improve user experience.

[0039] In the following, for the sake of ease of understanding and explanation, the execution process and actions of the real-time communication method of the present application in a video conferencing system are explained as an example rather than a limitation.

[0040] The following is combined with Figure 1The overall structure of a video conferencing system provided by an embodiment of the present application is exemplified.

[0041] like Figure 1 As shown, a video conferencing system provided by an embodiment of the present application includes: multiple user terminals, a signaling server and a media forwarding server.

[0042] In an embodiment of the present application, the signaling server may also be referred to as a management terminal or a control terminal. The signaling server is used to perform signaling interaction with a user terminal to transmit the necessary information required for establishing a connection between user terminals. At the same time, the signaling server is used to obtain the subscription relationship of the user terminal, and based on the subscription relationship, send instructions to the user terminal and the media forwarding server to determine the producer and consumer, and instruct the media forwarding server to send the resolution of the video stream to the user terminal through instructions. The role of the signaling server in the embodiment of the present application is to act as an intermediary to help both parties establish a connection with as little privacy exposure as possible. The communication method between the signaling server and the user terminal and the media forwarding server can use any method, for example, a network socket (WebSocket) or an extensible markup language hypertext transfer protocol request (XMLHttpRequest) to exchange each other's token information. The embodiment of the present application is not limited to this.

[0043] It should be understood that the media forwarding server can also be called a selective forwarding unit (Selective Forwarding Unit, SFU) or a forwarding server. If there is no special explanation below, the several names of the media forwarding server can be replaced, and the embodiments of the present application are not limited to this. It should be understood that in the present application, the signaling server and the SFU can be two independent devices or integrated into one device. In the embodiments of the present application, the media forwarding server can be Mediasoup, Janus, Jitsi, Kurento or Medooze, etc. The embodiments of the present application are not limited to this. The following only takes SFU as Mediasoup as an example for explanation. The media forwarding server can accept the instructions of the signaling server to manage and control the producer and consumer objects, and then decide whether to receive and / or send the audio and / or video streams of the real-time communication of the user terminal.

[0044] It should be understood that the user terminal in the embodiment of the present application can also be referred to as a user terminal or user equipment or terminal equipment or client. In the real-time example of the present application, the terminal equipment can be installed with a browser, and real-time communication can be performed through the browser, or an APP or a small program can be installed, and real-time communication can be performed through the APP or the small program. The terminal equipment in the present application may include a smart phone, a tablet computer, (personal digital assistant, PDA personal digital assistant), a computer, a game console, a wearable device, a tablet computer (portable android device, PAD), etc., and the embodiment of the present application is not limited thereto. The user terminal in the embodiment of the present application can send a video stream to the media forwarding server as a producer, or can receive a video stream sent by the media forwarding server as a consumer. Among them, the user terminal as a producer, the video stream sent may include multiple resolutions, for example, it may include a first resolution and a second resolution as described below, wherein the first resolution may include a first sub-resolution and a second sub-resolution. The video stream received by the terminal device as a consumer can be sent by the terminal device through a signaling server to instruct the media forwarding server to send a video stream of the corresponding resolution according to the mode displayed by the terminal device.

[0045] It should be understood that the operating system running on the electronic device can be a mobile version of Android, Ubuntu mobile version, Tizen and other Linux kernel-based operating systems as well as desktop operating systems such as Windows, Mac OS, Linux, etc., but the present invention is not limited to this.

[0046] In the embodiments of the present application, the terminal device may refer to an access terminal, a user equipment (UE), a user unit, a user station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent or a user device. The access terminal may be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, a terminal device in a 5G network or a terminal device in a future evolved Public Land Mobile Network (PLMN), etc.

[0047] It should be understood that Figure 1In the video conferencing system scenario shown in , only two user terminals are illustrated, but the embodiments of the present application are not limited to this. In an actual real-time communication scenario, the number of user terminals can include any number, depending on the actual situation, for example, it can be 10, 50, 500, 5000, etc.

[0048] The following is combined with Figure 2 A method for transmitting data in a video conference provided by an embodiment of the present application is exemplarily described. Figure 2 In the method shown, the video producer will upload video streams of multiple resolutions to the media forwarding server at the same time, such as the video streams of the first resolution and the second resolution mentioned above. The first resolution can correspond to the main view mode, and the second resolution can correspond to the gallery mode. The first resolution is higher than the second resolution. For consumers' video viewers, the main view model is full screen, and high-resolution videos will be loaded. The gallery mode is a small view, and low-resolution videos will be loaded.

[0049] Specifically, Figure 2 The method shown is applied to Figure 1 In the video conferencing system shown, Figure 2 The methods shown include:

[0050] 210. The user terminal generates a subscription message.

[0051] The subscription message includes an identifier of a producer subscribed by a user terminal, and resolution indication information of a subscribed video stream, wherein the resolution indication information is used to indicate the resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the plurality of preset resolutions including a first resolution and a second resolution, and the first resolution is greater than the second resolution.

[0052] It should be understood that in the embodiment of the present application, one user terminal corresponds to one member who joins the meeting, and the member who joins the meeting joins the meeting through the user terminal. There are two channels between a user terminal and the SFU, including an uplink channel and a downlink channel. The uplink channel is used for the user terminal to upload a video stream and / or an audio stream to the SFU, and the downlink channel is used for the SFU to transmit the video stream and / or the audio stream of other members who join the meeting to the user terminal. The producer identifier can be the terminal identifier of the user terminal or the identifier of the member who joins the meeting, and the embodiment of the present application is not limited to this. Optionally, the subscription message can also be a consumer identifier, and the consumer identifier can be the terminal identifier of the user terminal or the identifier of the member who joins the meeting, and the embodiment of the present application is not limited to this. If a member who joins the meeting A subscribes to the audio of another member who joins the meeting B, the member A is an audio consumer, and if the member A subscribes to the video of the member B, the member A is a video consumer. Since members cannot become their own consumers, if the number of members who join the meeting is N, then for a member, it can become a maximum of 2N-2 consumers.

[0053] In the embodiment of the present application, the subscription message may be generated when the user joins the meeting or during the meeting. For example, it is generated when the user slides the page or cancels or opens the video screen of a certain user. In the embodiment of the present application, the subscription message is mainly generated when the user enters the video conference and loads the meeting for the first time, or adjusts the video screen during the meeting. Since it involves loading the meeting screen or changing the meeting screen, the user needs to operate to tell the signaling server the resolution of the subscribed video stream.

[0054] Correspondingly, as an embodiment, in 210, generating the subscription message includes: generating the subscription message according to the user's operation instruction, and the user's operation instruction includes an operation instruction for entering a video conference or an operation instruction for adjusting a video screen. The operation instruction for adjusting a video screen includes a sliding operation instruction, a clicking operation instruction, a view switching operation instruction, a gesture operation instruction, a terminal shaking operation instruction, or a voice control operation instruction.

[0055] Specifically, the video conference entry operation instruction may be an operation of a user opening a conference, joining a conference, entering a conference, or opening a user's video screen during a conference. The video screen adjustment operation may be any operation for switching pages or adjusting the screen. It may be an operation of the user on the screen, a gesture operation captured by a camera, an eye operation, or a voice control operation.

[0056] For example, the user can operate the interface displayed on the user terminal. The specific user operation form can be flexibly operated according to the form of the user terminal. For example, when the terminal is a mobile phone or a PC, the specific operation form can be adapted and adjusted. The embodiments of the present application are not limited to this. For example, the user can subscribe to members by turning pages. For another example, it can be an operation of adjusting the display view mode or quantity by the user. For example, before the adjustment, there are 9 views, and after the user operation, it is adjusted to a 16-view mode.

[0057] The following takes switching between the main view mode and the gallery mode, and turning pages in the gallery mode as examples for explanation.

[0058] Optionally, as an embodiment, the current mode is the main view mode, and the user's operation instruction is used to switch the current model from the main view mode to the gallery mode, and the resolution of the video stream subscribed by the user terminal is the second resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0059] Optionally, as another embodiment, the current mode is a gallery mode, the user's operation instruction is used to turn pages in the gallery mode, and the resolution size of the video stream subscribed by the user terminal is the second resolution.

[0060] Optionally, as another embodiment, the current mode is the gallery mode, and the operation instruction is used to switch the current model to the main view mode. The resolution of the video stream subscribed by the user terminal is the first resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0061] Specifically, when switching from the main view mode to the gallery mode, the resolution of the video stream in the subscription message is the second resolution. During the switching process, the main view screen with the first resolution and a page screen with the second resolution under the gallery model are loaded simultaneously. When switching from the gallery mode to the main view mode, the resolution of the video stream in the subscription message is the first resolution. During the switching process, a page screen with the second resolution under the gallery model and the main view screen with the first resolution are loaded simultaneously. When switching pages in the gallery mode, the resolution of the video stream in the subscription message is the second resolution. During the switching process, the previous page screen with the second resolution under the gallery model and the next page screen after switching are loaded simultaneously.

[0062] Optionally, as another embodiment, the first resolution includes a first sub-resolution or a second sub-resolution, wherein the first sub-resolution is greater than the second sub-resolution, and both the first sub-resolution and the second sub-resolution are greater than the second resolution, and the network status of the user terminal corresponding to the first sub-resolution is better than the network status of the user terminal corresponding to the second sub-resolution.

[0063] Specifically, when it comes to a subscription message including a main view screen of the first resolution, the subscription message may have two situations. The first situation is when the network status is good, for example, the network status is greater than or equal to the network status threshold, and the value of the first resolution is the first sub-resolution; the second situation is when the network status is poor, for example, the network status is less than the network status threshold, and the value of the first resolution is the second sub-resolution.

[0064] It should be understood that the above description is that the subscription message is generated based on the user's operation. Optionally, the subscription message may also be automatically generated by the user terminal. For example, when the user terminal automatically turns the page, the subscription message may be generated by the user terminal according to the page mode after turning the page. The embodiments of the present application are not limited to this.

[0065] 220. The user terminal sends the subscription message to the signaling server.

[0066] Correspondingly, the signaling server receives the subscription message.

[0067] 230. The signaling server generates a subscription indication message according to the subscription message.

[0068] The subscription indication message is used to instruct the media forwarding server to send a video stream of corresponding resolution to the user terminal.

[0069] Specifically, the subscription indication message may include the resolution of the video stream. The subscription indication message may be used to instruct the media forwarding server to send the video stream of the corresponding resolution of the producer to the consumer. Optionally, the subscription indication message may also include the identifier of the producer and the identifier of the consumer consuming the producer.

[0070] 240. The media forwarding server sends the video stream to the user terminal.

[0071] Correspondingly, the user terminal receives the video stream sent by the media forwarding server. Specifically, after receiving the subscription indication message, the media forwarding server sends the producer's video stream of the corresponding resolution to the user terminal (ie, the consumer) according to the resolution in the subscription indication message.

[0072] That is to say, during the user's operation or during the picture adjustment process, the embodiment of the present application will newly load the video stream of the adjusted picture while continuing to load the video stream before the adjustment. Further, after the operation is completed or the picture adjustment is completed, the embodiment of the present application may also include canceling the subscription to the video stream before the adjustment.

[0073] Optionally, as another embodiment, after the action of adjusting the video screen corresponding to the user's operation instruction is completed, the method further includes: sending a cancellation subscription message to the signaling server, so that the signaling server generates a cancellation indication message according to the cancellation subscription message, and the indication message is used to instruct the media forwarding server to cancel sending the video stream before the screen is adjusted to the user terminal, wherein the cancellation subscription message includes an identifier of the video stream before the screen is adjusted.

[0074] Specifically, when the screen after switching is different from the screen before switching and does not overlap, the embodiment of the present application can cancel the loading of the screen before switching by canceling the subscription.

[0075] Optionally, when the screen after switching overlaps with the screen before switching, for example, there is no main view screen before switching, and after switching, it is gallery mode and the main view screen before switching is included, the embodiment of the present application may not send a cancellation subscription message after switching, but may send a resolution adjustment request to reduce the resolution of the original main view screen. Similarly, after switching from gallery mode to main view, if the main view screen is a screen in the original gallery mode, the embodiment of the present application may not send a cancellation subscription message after switching, but may send a resolution adjustment request to increase the resolution of the main view screen.

[0076] Optionally, as another embodiment, in the embodiment of the present application, the unsubscribe message or the resolution adjustment request may not be sent, and after the screen is switched, a new subscription message may be sent, the subscription message including the video stream to be loaded and the resolution information. That is, the subscription message is used to indicate the latest subscription status of the signaling server.

[0077] In the embodiment of the present application, the video producer uploads video streams of multiple resolutions, and the consumer sends corresponding subscription messages to the signaling server according to different needs, so that the consumer can flexibly load video streams of different resolutions according to different display view modes. Since the size of a picture displayed in the main view mode and the gallery mode is different, in order to obtain the same viewing effect, the requirements for resolution are also different. In the embodiment of the present application, different resolution video streams are loaded through different modes, and the full-screen main view and the small view of the gallery are loaded with video streams of different resolutions through the video resolution lifting layer technology, thereby reducing the bandwidth occupation. Compared with the prior art that all video streams are loaded with a fixed resolution, it can save resource overhead and reduce resource waste. At the same time, by subscribing to the video, the embodiment of the present application ensures the fluidity of the loading when switching between the main view and the gallery mode, and cancels the subscription of the picture before the switch after the switch, thereby saving unnecessary performance consumption.

[0078] Combined with the previous article Figure 1 to Figure 2 The method for transmitting data in a video conference in an embodiment of the present application is described below. Figure 3 and Figure 4 Describe the device for transmitting data in a video conference in an embodiment of the present application, combined with Figure 5 An electronic device according to an embodiment of the present application is described.

[0079] Please refer to Figure 3 , Figure 3 A schematic diagram of a device for transmitting data in a video conference provided by an embodiment of the present application is shown. Figure 3 The device 300 shown can be a user terminal. It should be understood that the device 300 corresponds to the user terminal in the above method embodiment, and can execute the various steps performed by the user terminal involved in the above method embodiment. The specific functions of the device 300 can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0080] Figure 3 The device 300 shown includes at least one software function module that can be stored in a memory or fixed in the device in the form of software or firmware. Figure 3 The device 300 shown comprises:

[0081] A generating unit 310 is configured to generate a subscription message, wherein the subscription message includes an identifier of a producer subscribed by a user terminal and resolution indication information of a subscribed video stream, wherein the resolution indication information is used to indicate a resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions, wherein the plurality of preset resolutions includes a first resolution and a second resolution, and the first resolution is greater than the second resolution;

[0082] A sending unit 320 is configured to send the subscription message to a signaling server, so that the signaling server generates a subscription indication message according to the subscription message, wherein the subscription indication message is used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal;

[0083] The receiving unit 330 is configured to receive the video stream sent by the media forwarding server.

[0084] In one embodiment, the generating unit is specifically used to generate the subscription message according to the user's operation instruction, and the user's operation instruction includes an operation instruction for entering a video conference or an operation instruction for adjusting a video screen. The operation instruction for adjusting a video screen includes: a sliding operation instruction, a clicking operation instruction, a view switching operation instruction, a gesture operation instruction, a shaking terminal operation instruction, or a voice control operation instruction.

[0085] In one embodiment, the current mode is the main view mode, and the user's operation instruction is used to switch the current model from the main view mode to the gallery mode. The resolution of the video stream subscribed by the user terminal is the second resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0086] In one implementation, the current mode is a gallery mode, the user's operation instruction is used to turn pages in the gallery mode, and the resolution of the video stream subscribed by the user terminal is the second resolution.

[0087] In one embodiment, the current mode is the gallery mode, and the operation instruction is used to switch the current model to the main view mode. The resolution of the video stream subscribed by the user terminal is the first resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

[0088] In one embodiment, after the action of adjusting the video screen corresponding to the user's operation instruction is completed, the sending unit is also used to send a cancellation subscription message to the signaling server, so that the signaling server generates a cancellation indication message according to the cancellation subscription message, and the indication message is used to instruct the media forwarding server to cancel sending the video stream before the screen is adjusted to the user terminal, wherein the cancellation subscription message includes an identifier of the video stream before the screen is adjusted.

[0089] In one embodiment, the first resolution includes a first sub-resolution or a second sub-resolution, wherein the first sub-resolution is greater than the second sub-resolution, and both the first sub-resolution and the second sub-resolution are greater than the second resolution, and the network status of the user terminal corresponding to the first sub-resolution is better than the network status of the user terminal corresponding to the second sub-resolution.

[0090] Please refer to Figure 4 , Figure 4 A schematic diagram of a device for transmitting data in a video conference provided by an embodiment of the present application is shown. Figure 4 The device 400 shown can be a signaling server. It should be understood that the device 400 corresponds to the signaling server in the above method embodiment, and can execute each step performed by the signaling server involved in the above method embodiment. The specific functions of the device 400 can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0091] Figure 4 The device 400 shown includes at least one software function module that can be stored in a memory or fixed in the device in the form of software or firmware. Figure 4 The device 400 shown includes: a receiving unit 410, used to receive a subscription message sent by a user terminal, the subscription message including an identifier of a producer subscribed by the user terminal, and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate the resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions, the plurality of preset resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; a generating unit 420, used to generate a subscription indication message according to the subscription message, the subscription indication message being used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal; and a sending unit 430, used to send the subscription indication message to the media forwarding server.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0093] like Figure 5 As shown, an embodiment of the present application provides an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program to implement the method executed by the signaling server or the method executed by the user terminal in any of the above embodiments. Optionally, Figure 5 The device shown may also include a transceiver, which may be used to send and / or receive data streams. Figure 5 The device shown may also include a collector, which may be used to collect audio / video data. The optional electronic device 500 may also include other input devices or other hardware devices, etc., but the embodiments of the present application are not limited thereto.

[0094] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0095] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some functions or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the disclosed embodiment may be used to execute instructions in the memory 510 to implement the method shown above to be performed by the signaling server or the user terminal. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0096] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement a method performed by a signaling server or a user terminal in any of the above methods provided in the above embodiments.

[0097] An embodiment of the present application further provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the method performed by a signaling server or a user terminal in any of the above methods provided in the above embodiments.

[0098] It should be noted that the processor in the embodiment of the present invention (eg, Figure 5The processor in the embodiment of the present invention may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiment may be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor may be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general processor may be a microprocessor or the processor may be any conventional processor, etc. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0099] It can be understood that the memory in the embodiment of the present invention (for example, Figure 5The memory in the memory may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] The application programs involved in the embodiments of the present application include any application installed on the requesting end, including but not limited to browsers, email, instant messaging services, word processing, keyboard virtualization, widgets, encryption, digital rights management, voice recognition, voice replication, positioning (such as functions provided by the global positioning system), music playback, etc.

[0101] It should be understood that the transceiver unit or transceiver in the embodiment of the present invention may also be referred to as a communication unit.

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0103] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0104] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0105] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0106] It should be understood that in the embodiment of the present invention, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0107] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0111] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] In short, the above is only a preferred embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for transmitting data in a video conference, characterized in that: include: Generate a subscription message, the subscription message including an identifier of a producer subscribed by a user terminal, and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate a resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the plurality of preset resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; Sending the subscription message to a signaling server so that the signaling server generates a subscription indication message according to the subscription message, wherein the subscription indication message is used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal; Receive the video stream sent by the media forwarding server.

2. The method according to claim 1, characterized in that The generating of the subscription message comprises: The subscription message is generated according to the user's operation instruction, where the user's operation instruction includes an operation instruction for entering a video conference or an operation instruction for adjusting a video screen; The video picture adjustment operation instruction includes: a sliding operation instruction, a clicking operation instruction, a view switching operation instruction, a gesture operation instruction, a terminal shaking operation instruction or a voice control operation instruction.

3. The method according to claim 2, characterized in that The current mode is the main view mode, and the user's operation instruction is used to switch the current model from the main view mode to the gallery mode. The resolution of the video stream subscribed by the user terminal is the second resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

4. The method according to claim 2, characterized in that: The current mode is the gallery mode, the user's operation instruction is used to turn pages in the gallery mode, and the resolution of the video stream subscribed by the user terminal is the second resolution.

5. The method according to claim 2, characterized in that: The current mode is the gallery mode, and the operation instruction is used to switch the current model to the main view mode. The resolution of the video stream subscribed by the user terminal is the first resolution, wherein the resolution of the video stream corresponding to the main view mode is the first resolution, and the resolution of the video stream corresponding to the gallery mode is the second resolution.

6. The method according to any one of claims 2 to 5, characterized in that After the action of adjusting the video screen corresponding to the user's operation instruction is completed, the method further includes: A cancellation subscription message is sent to the signaling server so that the signaling server generates a cancellation indication message according to the cancellation subscription message, wherein the indication message is used to instruct the media forwarding server to cancel sending the video stream before the picture is adjusted to the user terminal, wherein the cancellation subscription message includes an identifier of the video stream before the picture is adjusted.

7. The method according to any one of claims 2 to 5, characterized in that The first resolution includes a first sub-resolution or a second sub-resolution, wherein the first sub-resolution is greater than the second sub-resolution, and both the first sub-resolution and the second sub-resolution are greater than the second resolution, and the network status of the user terminal corresponding to the first sub-resolution is better than the network status of the user terminal corresponding to the second sub-resolution.

8. A method for transmitting data in a video conference, characterized in that: include: Receiving a subscription message sent by a user terminal, the subscription message including an identifier of a producer subscribed by the user terminal and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate a resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the plurality of preset resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; Generate a subscription indication message according to the subscription message, wherein the subscription indication message is used to instruct the media forwarding server to send a video stream of a corresponding resolution to the user terminal; Send the subscription indication message to the media forwarding server.

9. A device for transmitting data in a video conference, characterized in that: include: a generating unit, configured to generate a subscription message, wherein the subscription message includes an identifier of a producer subscribed by a user terminal and resolution indication information of a subscribed video stream, wherein the resolution indication information is used to indicate a resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, wherein the plurality of preset resolutions includes a first resolution and a second resolution, and the first resolution is greater than the second resolution; A sending unit, configured to send the subscription message to a signaling server, so that the signaling server generates a subscription indication message according to the subscription message, wherein the subscription indication message is used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal; The receiving unit is used to receive the video stream sent by the media forwarding server.

10. A device for transmitting data in a video conference, characterized in that: include: a receiving unit, configured to receive a subscription message sent by a user terminal, the subscription message including an identifier of a producer subscribed by the user terminal and resolution indication information of a subscribed video stream, the resolution indication information being used to indicate a resolution of the video stream subscribed by the user terminal, wherein the resolution of the video stream subscribed by the user terminal is one of a plurality of preset resolutions corresponding to a display view mode, the plurality of preset resolutions including a first resolution and a second resolution, the first resolution being greater than the second resolution; A generating unit, configured to generate a subscription indication message according to the subscription message, wherein the subscription indication message is used to instruct a media forwarding server to send a video stream of a corresponding resolution to the user terminal; A sending unit is used to send the subscription indication message to the media forwarding server.

Images