Method, apparatus, device, medium and product for processing layout of media stream
Patent Information
- Application Number
- CN202510458960.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2026-10-20
Smart Images
Figure BDA0005356961100000101 
Figure HDA0005356961160000011 
Figure HDA0005356961160000021
Abstract
Description
Technical Field
[0001] The embodiments of this disclosure generally relate to the fields of audio and video processing and live streaming technology, and specifically to methods, apparatus, devices, media, and products for processing the layout of media streams. Background Technology
[0002] With the continuous development of audio and video technology and live streaming technology, the application scenarios of online live streaming, video conferencing, and remote interaction are increasing, and have become an important part of online communication today. Various social media platforms, online education systems, and remote office software are placing higher demands on high-quality media streaming and interactive experiences. In the live streaming industry, multi-user interaction has become a major development direction, and the real-time connections and interactions between hosts, viewers, and guests are becoming more diverse. This has led to many more flexible and intelligent solutions for video layouts in different scenarios.
[0003] With globalization, various live streaming and video conferencing systems need to support efficient layout management across multiple terminals and networks to ensure proper display of images and meet the broadcaster's desired effects, guaranteeing a smooth user experience. Simultaneously, with the widespread adoption of smart terminals and mobile networks, audio and video interaction experiences are continuously evolving towards greater clarity, lower latency, and a more immersive experience. Summary of the Invention
[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and product for processing the layout of media streams.
[0005] According to a first aspect of this disclosure, a method for processing the layout of a media stream is provided. The method includes sending a request associated with a live stream from a client to a server to receive from the server a layout identifier corresponding to a target layout for a set of media streams from a group of users in the live stream. The method further includes, in response to the existence of a set of configuration template files for the layout of the media streams for the live stream on the client, determining, based on the layout identifier, whether the set of configuration template files includes the target layout. The method further includes, in response to the target layout not being included in the set of configuration template files, sending a template retrieval request to the server to receive from the server a target configuration template file including the layout identifier, the template retrieval request including the layout identifier. The method further includes, based on the target configuration template file, displaying the set of media streams from the group of users in the live stream using the target layout corresponding to the layout identifier.
[0006] According to a second aspect of this disclosure, a method for processing the layout of a media stream is provided. The method includes, in response to receiving a request associated with a live broadcast from a client at a server, sending a layout identifier corresponding to a target layout for a set of media streams from a group of users in the live broadcast to the client. The method further includes, in response to a set of configuration template files for the layout of the media streams for the live broadcast in the client, receiving a template retrieval request from the client, the template retrieval request including the layout identifier. The method further includes searching for a target configuration template file including the layout identifier from a plurality of configuration template files. The method further includes sending the target configuration template file to the client for displaying a set of media streams from a group of users in the live broadcast.
[0007] According to a third aspect of this disclosure, an apparatus for processing the layout of media streams is provided. The apparatus includes a layout identifier receiving module configured to send a request associated with a live broadcast from a client to a server to receive from the server a layout identifier corresponding to a target layout for a group of media streams belonging to a group of users in the live broadcast; a template matching module configured to, in response to the existence of a set of configuration template files for the layout of the media streams in the live broadcast on the client, determine whether the target layout is included in the set of configuration template files based on the layout identifier; a template acquisition module configured to, in response to the target layout not being included in the set of configuration template files, send a template acquisition request to the server to receive from the server a target configuration template file including the layout identifier, the template acquisition request including the layout identifier; and a layout display module configured to, based on the target configuration template file, display the group of media streams belonging to a group of users in the live broadcast using the target layout corresponding to the layout identifier.
[0008] According to a fourth aspect of this disclosure, an apparatus for processing the layout of media streams is provided. The apparatus includes a layout identifier sending module configured to, in response to receiving a request associated with a live broadcast from a client at a server, send a layout identifier corresponding to a target layout of a group of media streams belonging to a group of users in the live broadcast to the client; a template retrieval request receiving module configured to, in response to the target layout corresponding to the layout identifier not being included in a set of configuration template files for the layout of the media streams for the live broadcast in the client, receive a template retrieval request from the client, the template retrieval request including the layout identifier; a lookup module configured to look up a target configuration template file including the layout identifier from a plurality of configuration template files; and a configuration template file sending module configured to send the target configuration template file to the client for displaying a group of media streams belonging to a group of users in the live broadcast.
[0009] In a fifth aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the methods according to the first and second aspects of this disclosure.
[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.
[0011] In a seventh aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.
[0012] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0014] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;
[0015] Figure 2 The illustration shows a schematic diagram of an example method for laying out a media stream according to some embodiments of the present disclosure;
[0016] Figure 3 The illustration shows a schematic diagram of another example method for the layout of a media stream processing according to some embodiments of the present disclosure;
[0017] Figure 4 The illustration shows a schematic diagram of an example method for laying out a media stream according to some embodiments of the present disclosure;
[0018] Figure 5 The illustration shows schematic diagrams of different layouts for four-person live streaming according to some embodiments of the present disclosure;
[0019] Figure 6 The illustration shows schematic diagrams of different layouts for six-person live streaming according to some embodiments of the present disclosure;
[0020] Figure 7The illustration shows schematic diagrams of different layouts for nine-person live streaming according to some embodiments of the present disclosure;
[0021] Figure 8 The illustration shows a schematic diagram of an example of dynamically adjusting a layout according to some embodiments of the present disclosure;
[0022] Figure 9 The illustration shows a schematic block diagram of an apparatus for processing media streams according to some embodiments of the present disclosure;
[0023] Figure 10 The illustration shows a schematic block diagram of another apparatus for processing media streams according to some embodiments of the present disclosure;
[0024] Figure 11 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated. Detailed Implementation
[0025] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0026] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0027] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0028] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0029] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0031] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0032] As mentioned above, with the rapid development of the mobile internet and live streaming industry, the demand for interactive live streaming on various social live streaming platforms, online conferencing software, and online learning software is increasing daily. Currently, live streaming platforms allow multiple users to connect simultaneously and offer diverse video layouts to enhance the user experience. In scenarios such as social entertainment, remote work, and online classes, a well-designed live streaming layout can optimize the user's viewing experience and make the presented content clearer. Furthermore, with the development of technologies such as cloud computing, these technologies provide stronger computing power for the dynamic adjustment of audio and video layouts, enabling smoother rendering and presentation of multi-user video feeds across different terminal devices.
[0033] While existing live streaming co-op solutions can present a wide range of video layouts, they still have certain limitations. For example, in current co-op layout solutions, the maximum number of co-op participants and the layout style are fixed. When changes are needed to the maximum number of co-op participants or the layout style, the code needs to be modified and users need to upgrade to a new client. This not only increases development and maintenance costs but also affects the user experience.
[0034] To address this, embodiments of this disclosure propose a method for processing media stream layouts. In this method, a request associated with the live stream is sent from the client to the server to receive a layout identifier corresponding to a target layout for a group of media streams from a group of users in the live stream. If a set of configuration template files for the layout of the media streams in the live stream exists on the client, the layout identifier is further used to determine whether the set of configuration template files includes the target layout. If the target layout is not included in the set of configuration template files, the client needs to send a template retrieval request to the server to receive a target configuration template file including the layout identifier, where the template retrieval request includes the layout identifier. Finally, the client can use the target layout corresponding to the layout identifier in the target configuration template file to display a group of media streams from a group of users in the live stream. This method uses the layout and layout identifier in the configuration template file to display the client's live stream layout, allowing the client to update the layout simply by downloading the configuration template file, quickly adapting to the needs of different scenarios and improving the user experience.
[0035] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1 An example environment is shown in which the devices and / or methods of embodiments of this disclosure may be implemented. Environment 100 includes a broadcaster 102, a client 104, and a server 106.
[0036] Examples of broadcaster 102, client 104 and server 106 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.
[0037] like Figure 1 As shown, in a live streaming scenario, the broadcaster's client 102 can start a live stream and establish conversations with other users. When the broadcaster starts a multi-user conversation through the broadcaster's client 102, they can select layout parameters corresponding to the users participating in the live stream, such as the layout style to use and the maximum number of users participating in the conversation. For example, the broadcaster can choose a grid layout with a maximum of 8 users. After the broadcaster's client 102 provides the above parameters to the server 106, the server 106 can select a corresponding layout based on these parameters and determine the layout identifier 114 of that layout. For ease of description, this layout can be called the first layout, and the layout identifier 114 can also be called the first layout identifier. The server 106 stores the identifier of the live stream and the corresponding layout identifier 114 in association. For example, if the live stream is conducted in a live streaming room, the identifier of the live stream is the identifier of the live streaming room.
[0038] When a viewer or guest accesses a live stream via client 104, they can send a request 108 associated with the live stream to server 106. Client 104 can then obtain a layout identifier 114 for that live stream from server 106. For example, client 104 can send the live stream's identifier to server 106 to obtain the layout identifier 114 from server 106.
[0039] Traditional clients embed the live stream layout into the client itself, and client 104 can only use the pre-defined layout to display the video stream layout of the users interacting with the live stream. However, in this disclosure, client 104 does not embed the layout into the application; instead, it can download a set of configuration template files 116, which include different layouts. Therefore, client 104 can determine the live stream layout by combining the set of configuration template files 116 with the layout identifier 114 obtained from server 106. This set of configuration template files 116 may include one configuration template file or multiple configuration template files. In one example, this set of configuration template files is loaded from the server during a cold start of the live stream application. In another example, this set of configuration template files is loaded from the server when the application was previously used. The above examples are merely for describing this disclosure and are not intended to limit it.
[0040] In addition, to adapt to the rapidly changing user needs of live streaming, developers typically provide many new live chat layouts on the server-side (106) for users to use. Therefore, the server-side (106) can provide new layout parameters to the broadcaster for selection. If the broadcaster (102) selects a new layout parameter, the layout corresponding to that parameter may not exist in a set of configuration template files on the client. If the broadcaster (102) selects an older layout parameter, the layout corresponding to that parameter may exist in a set of template configuration files on the client. To accurately display the live chat layout, the client needs to match the layout identifier (114) obtained from the client with the layout identifier in a set of configuration template files (116). If the layout identifier is found, the first layout corresponding to that layout identifier is retrieved from the set of configuration template files (116) to display a media stream (118) on the client (104).
[0041] If layout identifier 114 is a new layout identifier for client 104, then this layout identifier will not be found in a set of configuration files of client 104. In this case, client 104 needs to generate a template retrieval request 120 including layout identifier 114 to send to server 106. After receiving the template retrieval request 120, server 106 will search for a target configuration template file 110 containing this layout identifier in multiple configuration template files stored on the server. Then, server 106 sends the target configuration template file 110 to client 104, so that client 104 can re-parse the target layout 112 to be used based on the target configuration template file. Then, the target layout is used to display a set of media streams.
[0042] In live streaming scenarios, if the target layout cannot be changed after initial setup, the user experience will deteriorate when the number of connected users changes or user needs evolve. For example, if the initial maximum number of connected users is set to 4, but the streamer finds that the number of users joining the connection far exceeds the initial setting, other users will be unable to join. To further mitigate this issue, this disclosure allows for real-time dynamic adjustment of the connection layout. For instance, if a grid layout with a maximum of 4 connected users is initially set, it can be changed to a panel layout with a maximum of 9 connected users due to an increase in users, thus dynamically adjusting the connection layout. Therefore, it is necessary to continue providing the streamer with layout parameter selection options during the live stream. If the streamer adjusts the corresponding layout parameters, the server 106 can determine the layout identifier corresponding to the adjusted layout parameters after receiving the adjustment. This layout identifier is then sent to the client in real time, and the client 104 can further check a set of configuration template files and / or newly downloaded layout configuration files to adjust the layout.
[0043] In addition, since this disclosure uses a set of configuration template files, a configuration template file parsing engine needs to be loaded on the client. Both the set of configuration template files obtained by client 104 from server 106 and the newly downloaded configuration template files need to be parsed and processed by this parsing engine.
[0044] Because the software applications are installed at different times on different clients, earlier versions may not have a parsing engine for configuration template files installed. In this case, even if the client receives the layout identifier, it cannot obtain a usable configuration template file because the client lacks the parsing engine. To solve this problem, client 104 can further send a request to server 106 to obtain the parsing engine. Upon receiving this request, server 106 will obtain the parsing engine. Additionally, server 106 will also obtain the configuration template file corresponding to the layout identifier. Then, it will send the parsing engine and the configuration template file together to client 104. Client 104 then installs the parsing engine and uses it to parse the configuration template file to obtain the corresponding target layout for displaying the video stream layout of the connected user. Alternatively, Figure 1 The client 104 and the broadcaster 102 shown can be the same device, that is, the broadcaster can be used as the client.
[0045] This method uses the layout and layout identifiers in the configuration template file to display the client's live chat layout. This allows the client to adjust the layout simply by downloading the configuration template file, quickly adapting to the needs of different scenarios and improving the user experience. Furthermore, the layout can be dynamically adjusted, further enhancing the user experience.
[0046] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which devices and / or methods according to some embodiments of the present disclosure may be implemented, in conjunction with... Figure 2 A schematic diagram illustrating an example method for processing the layout of a media stream according to some embodiments of the present disclosure. Figure 2 The method in can be derived from Figure 1 It can be executed on client 104 or any suitable computing device.
[0047] like Figure 2 In example method 200, at block 202, a request associated with the live stream is sent from client 104 to server 106 to receive a layout identifier corresponding to the target layout of a group of media streams for a group of users in the live stream. This layout identifier can be obtained from server 106 when a user wants to access the live stream. For example, a user can obtain the layout identifier from the server when they want to watch or participate in the live stream. The layout identifier is determined based on the layout style received from the broadcaster and the maximum number of users participating in the live stream conversation. Alternatively, client 104 can be the broadcaster; therefore, the request in this case also includes the layout style and the maximum number of users participating in the live stream conversation.
[0048] At box 204, in response to the client having a set of configuration template files for the layout of the live media stream, client 104 determines whether the set of configuration template files includes the target layout based on the layout identifier. When the client has configuration template files, client 104 can check whether multiple layout identifiers in the set of configuration template files include the layout identifier for the live room received from the server. If the layout identifier is not present among the multiple layout identifiers, client 104 can determine that the set of configuration template files does not include the target layout.
[0049] At box 206, in response to the target layout not being included in a set of configuration template files, client 104 sends a template retrieval request to server 106 to receive a target configuration template file including a layout identifier from the server. The template retrieval request includes the layout identifier. At this point, in order to retrieve the layout corresponding to that layout identifier, client 104 will further request a target configuration template file including that layout identifier from server 106.
[0050] At box 208, client 104 uses a target layout corresponding to a layout identifier to display a set of media streams from a group of users in the live stream, based on the target configuration template file. After client 104 obtains the target configuration template file 110, it can retrieve the target layout corresponding to the layout identifier from it, and then use the target layout to display a set of media streams from a group of users connected to the live stream. For ease of description, this layout identifier can be referred to as the first layout identifier, and the target layout can be referred to as the first layout.
[0051] In some implementations, for example, if the broadcaster adjusts the target layout during a live stream, client 104 will receive a second layout identifier corresponding to the adjusted target layout from server 106. Client 104 can then determine whether a set of configuration template files and a target configuration template file include the second layout identifier. If both the set of configuration template files and the target configuration template file include the second layout identifier, the second layout corresponding to the second layout identifier can be obtained from both files. The second layout is then used to adjust the display of a set of media streams for a group of users in the live stream. For ease of description, the aforementioned template retrieval request is a first template retrieval request, and the target configuration template file is referred to as the first configuration template file.
[0052] Additionally, if a set of configuration template files and the first configuration template file do not include a second layout identifier, it indicates that a second layout corresponding to the second layout identifier does not exist in the client. In this case, client 104 needs to send a second template retrieval request to server 106 to receive a second configuration template file including the second layout identifier from server 106, wherein the second template retrieval request includes the second layout identifier. After client 104 obtains the second configuration template file from server 106, it uses the second layout in the second configuration template file to adjust the display of a set of media streams for a group of users in the live broadcast.
[0053] If, during the preceding detection process, client 104 detects that the layout identifier is included in a third configuration template file within a set of configuration template files, then client 104 can obtain the target layout corresponding to the layout identifier from the third configuration template file. Client 104 then further utilizes the target layout to display a set of media streams from a group of users in the live stream.
[0054] In some embodiments, when displaying a set of media streams for a group of users in a live stream using a target layout corresponding to a layout identifier, the newly acquired target configuration template file needs to be parsed first using a local parsing engine. Then, client 104 uses the target layout corresponding to the layout identifier in the parsed target configuration template file to display a set of media streams for a group of users in the live stream. Additionally, the set of configuration template files stored in client 104 is loaded during application cold start and parsed by the parsing engine.
[0055] Alternatively, if the client does not have a configuration template file for the layout of the live media stream, it indicates that the client does not have a parsing engine available for parsing the configuration template file. In this case, client 104 can send a request to server 106 to obtain the parsing engine. Then, client 104 can receive the parsing engine and a fourth configuration template file including layout identifiers from server 106. Thus, the client can use the parsing engine to parse the fourth configuration template file to obtain the target layout corresponding to the layout identifiers.
[0056] This method uses the layout and layout identifiers in the configuration template file to display the client's live chat layout. This allows the client to adjust the layout simply by downloading the configuration template file, quickly adapting to the needs of different scenarios and improving the user experience. Furthermore, the layout can be dynamically adjusted, further enhancing the user experience.
[0057] The above combination Figure 2 A schematic diagram illustrating an example method for laying out a media stream according to some embodiments of the present disclosure is described below, in conjunction with... Figure 3A schematic diagram illustrating another example method for the layout of a media stream processing according to some embodiments of the present disclosure. Figure 3 Example method 300 in the example can be derived from Figure 1 The server can be used for processing, either on a 106-bit machine or any suitable computing device.
[0058] At box 302, in response to receiving a request associated with the live stream from the client on the server side, a layout identifier corresponding to the target layout of a group of media streams for a group of users in the live stream is sent to the client. If the server 106 receives a join request from the client 104, it sends the layout identifier of the live stream room to the client 104.
[0059] At position 304, in response to the target layout corresponding to the layout identifier not being included in a set of configuration template files for the live media stream layout on the client, server 106 receives a template retrieval request from the client, the template retrieval request including the layout identifier. If the client 104 cannot find a layout corresponding to the layout identifier received from server 106, server 106 may further receive a request from the client to retrieve a configuration template file.
[0060] At 306, server 106 searches for a target configuration template file containing a layout identifier from multiple configuration template files. If server 106 receives a template retrieval request, it searches for a target configuration template file with that layout identifier from the multiple configuration template files already stored in server 106. Then, at 308, server 106 sends the target configuration template file to the client for displaying a set of media streams from a group of users in the live stream.
[0061] In some embodiments, when starting a live stream, the server 106 can receive layout parameter selections from the broadcaster's end, such as layout style and the maximum number of users participating in the live stream conversation. The server 106 can then determine the layout identifier based on the received layout parameters.
[0062] Furthermore, if adjusted layout parameters, such as layout style and maximum number of adjustments, are received from the broadcaster during the live stream, the server 106 can determine a second layout identifier corresponding to the adjusted target layout. The server 106 then sends the second layout identifier to the client 104 for displaying a set of media streams from a group of users in the live stream.
[0063] Additionally, server 106 may receive a second template retrieval request from client if a set of configuration template files and the first configuration template file do not include a second layout identifier. This second template retrieval request includes the second layout identifier. Then, server 106 further selects a second configuration template file corresponding to the second layout identifier from among multiple configuration template files. Next, server 106 sends the second configuration template file, including the second layout identifier, to client to adjust the display of a set of media streams for a group of users in the live stream using the second layout corresponding to the second layout identifier.
[0064] Furthermore, if the client does not have a configuration template file for the layout of the live media stream, the server 106 may receive a request from the client 104 to obtain the parsing engine. Then, the server 106 sends the parsing engine and a fourth configuration template file including layout identifiers to the client to obtain the target layout corresponding to the layout identifiers using the parsing engine.
[0065] This method uses the layout and layout identifiers in the configuration template file to display the client's live chat layout, allowing the server to adjust the layout simply by sending the configuration template file. This quickly adapts to the needs of different scenarios and improves the user experience. Furthermore, the layout can be dynamically adjusted, further enhancing the user experience.
[0066] The above combination Figure 3 A schematic diagram illustrating an example method for laying out a media stream according to some embodiments of this disclosure is provided below. Figure 4 A schematic diagram illustrating an example method for laying out a media stream according to some embodiments of the present disclosure. Figure 4 Example method 400 can be derived from Figure 1 The client 104 shown or any suitable computing device can execute this.
[0067] Example method 400 describes the process of handling the live stream layout based on a Domain Specific Language (DSL) configuration template. First, at box 402, the server maintains the latest DSL configuration template file, which can be used to define the rules for the live stream layout. When the client lacks a configuration template file for the live media stream layout or needs to update its local configuration template file, it will send a request to the server to retrieve the latest DSL template. This server can be in the cloud.
[0068] The DSL configuration template file is used to describe microphone information (number of microphones, location of microphones). For example, its format is as follows:
[0069] JSON
[0070] {
[0071] "max_number": 6 indicates that the maximum number of people allowed to connect in a live chat (maximum number of microphone slots) is 6.
[0072] "link_positions":[
[0073] {
[0074] "p1": Describes the location information of microphone position 1.
[0075] },
[0076]
[0077] At box 404, when the client application is launched, the client requests the DSL configuration template file from the server. After the server in the cloud successfully returns the DSL template, the client stores and parses it. At box 406, the client calls the DSL parsing engine to parse the downloaded DSL template. During the parsing process, the system reads key information defined in the DSL template, such as the live chat layout rules, the maximum number of users allowed in a live chat, and the location of each user.
[0078] Finally, at box 408, the client, based on the parsed DSL configuration information, can obtain the maximum number of connected users and their positions for that layout, according to the layout identifier of the user joining the live stream. At box 410, the client determines the connected user layout based on the maximum number of connected users and microphone position information. This layout can adapt to different live streaming scenarios, and it is one of several preset layouts selected by the broadcaster, such as an evenly divided grid layout or an unevenly divided layout.
[0079] This method, employing server-side DSL template configuration, client-side parsing, and dynamic layout adjustment, achieves more flexible and efficient layout management, providing a better user experience and technical support for scenarios requiring live streaming, video conferencing, and other scenarios that require multi-user interaction.
[0080] The above combination Figure 4 The following is a schematic diagram illustrating example methods of technical solutions for processing media stream layout according to some embodiments of the present disclosure. Figure 5 Schematic diagrams illustrating different layouts for four-person live streaming according to some embodiments of the present disclosure.
[0081] In Example 500, the live chat style of Example 502 supports a maximum of four people connecting via a grid, using an evenly divided grid format. When there are fewer than four users connecting, the microphone slots are evenly divided and occupy the live chat display area. When there are four users connecting, the microphone slots are evenly divided and occupy the display area, with each user's microphone slot appearing as a square. If another user requests to connect, they will enter a waiting queue until a microphone slot becomes available and the host approves their request. Example 504 demonstrates a non-evenly divided four-person live chat style, such as a panel format. In this case, the host's microphone slot occupies a larger area on the left, while users connecting via the grid evenly divide the smaller vertical area on the right.
[0082] The above combination Figure 5 The following is a schematic diagram illustrating different layouts for four-person live streaming according to some embodiments of the present disclosure, in conjunction with... Figure 6 Schematic diagrams illustrating different layouts for six-person live streaming according to some embodiments of the present disclosure.
[0083] In Example 600, the live chat style of Example 602 supports up to six people in a grid-based format. When there are six or fewer users on the microphone, all microphone positions are evenly divided into the live chat display area. Example 604 shows a non-uniformly divided six-person live chat style. In this case, the host's microphone position occupies a larger area on the left, while the users on the microphone share the smaller vertical area on the right.
[0084] The above combination Figure 6 The following is a schematic diagram illustrating different layouts for six-person live streaming according to some embodiments of the present disclosure, in conjunction with... Figure 7 Schematic diagrams illustrating different layouts for nine-person live streaming according to some embodiments of the present disclosure.
[0085] In Example 700, the live chat style of Example 702 supports up to nine people in a grid-based layout. When there are nine or fewer users with microphone positions, all microphone positions divide the live chat display area equally. Example 704 shows a non-equally divided nine-person live chat style. In this case, the host's microphone position occupies a larger area in the upper left corner, while the other users occupy the remaining live chat display area, presenting a layout where the host is in the upper left corner and the other microphones surround the host.
[0086] The above combination Figure 7 The following is a schematic diagram illustrating different layouts for six-person live streaming according to some embodiments of the present disclosure, in conjunction with... Figure 8 A schematic diagram illustrating examples of dynamically adjusting a layout according to some embodiments of the present disclosure.
[0087] In Example 800, we take the adjustment of the layout for a four-person live stream as an example. As the number of people wanting to join the live stream increases during the broadcast, and the host wants to change the display layout (802), the host will reselect layout parameters, such as the layout style and the maximum number of users participating in the conversation. When the host changes the layout style from an evenly divided grid to an unevenly divided grid and increases the maximum number of users, the server will send the corresponding unevenly divided layout identifier to the client. When the client receives this layout identifier, it will first determine if it can find the new layout. If the layout is found in the client's local configuration template text, the layout is dynamically adjusted. If the layout is not found in the client's local configuration template text, a request is sent to the server to obtain a new configuration template file to load the layout. After displaying the new layout, as shown in Layout 804, the area of the host's microphone position is increased, and the other five microphone positions are distributed in the vertical space to the right of the host after a preset adjustment, with the area occupied by the five microphone positions evenly distributed in the vertical space on the right.
[0088] Figure 9 The illustration shows a schematic block diagram of an apparatus for processing media streams according to some embodiments of the present disclosure. Figure 9 As shown, the apparatus 900 includes a layout identifier receiving module 902, configured to send a request associated with the live stream from a client to a server to receive from the server a layout identifier corresponding to a target layout for a group of media streams from a group of users in the live stream; a template matching module 904, configured to, in response to the existence of a set of configuration template files for the layout of the media streams in the live stream on the client, determine whether the target layout is included in a set of configuration template files based on the layout identifier; a template acquisition module 906, configured to, in response to the target layout not being included in a set of configuration template files, send a template acquisition request to the server to receive from the server a target configuration template file including the layout identifier, the template acquisition request including the layout identifier; and a layout display module 908, configured to, based on the target configuration template file, display a group of media streams from a group of users in the live stream using the target layout corresponding to the layout identifier.
[0089] In some embodiments, the layout identifier is determined based on the layout style received from the broadcaster's end and the maximum number of users participating in the live stream conversation.
[0090] In some embodiments, the layout identifier is a first layout identifier, the target layout is a first layout, and the apparatus 900 further includes: a second layout identifier receiving module configured to receive a second layout identifier corresponding to the adjusted target layout in response to the target layout being adjusted by the broadcaster during the live broadcast; a second layout obtaining module configured to obtain a second layout corresponding to the second layout identifier from a set of configuration template files and a target configuration template file in response to the second layout identifier being included in a set of configuration template files and a target configuration template file; and a first adjusting module configured to use the second layout to adjust the display of a set of media streams for a set of users in the live broadcast.
[0091] In some embodiments, where the template acquisition request is a first template acquisition request and the target configuration template file is a first configuration template file, the apparatus 900 further includes: a second template acquisition request sending module, configured to send a second template acquisition request to a server in response to a second layout identifier not being included in a set of configuration template files and a first configuration template file to receive a second configuration template file including the second layout identifier from the server, the second template acquisition request including the second layout identifier; and a second adjustment module, configured to adjust the display of a set of media streams for a set of users in a live broadcast based on the second layout in the second configuration template file.
[0092] In some embodiments, the apparatus 900 further includes: a target layout acquisition module, configured to acquire a target layout corresponding to the layout identifier from a third configuration template file in response to the layout identifier being included in a third configuration template file of a set of configuration template files; and a first media stream display module, configured to use the target layout to display a set of media streams of a set of users in a live broadcast.
[0093] In some embodiments, the layout display module 908 includes: a parsing module configured to parse a target configuration template file using a local parsing engine; and a second media stream display module configured to display a group of media streams from a group of users in a live broadcast using a target layout corresponding to a layout identifier in the parsed target configuration template file.
[0094] In some embodiments, the template matching module 904 includes: a layout identifier existence determination module, configured to determine whether a plurality of layout identifiers in a set of configuration template files exist; and a result determination module, configured to determine that a set of configuration template files does not include a target layout in response to the absence of a layout identifier among the plurality of layout identifiers.
[0095] In some embodiments, the apparatus 900 further includes: a parsing engine determination module, configured to determine that the client does not have a parsing engine that can be used to parse the configuration template file in response to the client not having a configuration template file for the layout of the live media stream; an acquisition request sending module, configured to send an acquisition request to the server for acquiring the parsing engine; a fourth configuration template file receiving module, configured to receive the parsing engine and a fourth configuration template file including a layout identifier from the server; and a second target layout acquisition module, configured to parse the fourth configuration template file using the parsing engine to obtain a target layout corresponding to the layout identifier.
[0096] In some embodiments, the template configuration file includes a layout style, a maximum number of users participating in the live chat, and the display position of the video stream for each user.
[0097] Figure 10 The illustration shows a schematic block diagram of another apparatus for processing media streams according to some embodiments of the present disclosure. Figure 10 As shown, the apparatus 1000 includes a layout identifier sending module 1002, configured to send a layout identifier corresponding to a target layout of a group of media streams for a group of users in the live stream to the client in response to receiving a request associated with live streaming from the client at the server; a template acquisition request receiving module 1004, configured to receive a template acquisition request from the client in response to the target layout corresponding to the layout identifier not being included in a set of configuration template files for the layout of the media streams for the live stream in the client, the template acquisition request including the layout identifier; a search module 1006, configured to search for a target configuration template file including the layout identifier from multiple configuration template files; and a configuration template file sending module 1008, configured to send the target configuration template file to the client for displaying a group of media streams for a group of users in the live stream.
[0098] In some embodiments, the apparatus 1000 further includes: a style and number receiving module configured to receive a layout style and a maximum number of users participating in the live broadcast from the broadcaster's end; and a layout identifier determining module configured to determine a layout identifier based on the layout style and the maximum number.
[0099] In some embodiments, the layout identifier is a first layout identifier, the target layout is a first layout, and the apparatus 1000 further includes: a second layout identifier determining module configured to determine a second layout identifier corresponding to the adjusted target layout in response to receiving at least one of an adjusted layout style and a maximum number of adjustments from the broadcaster during the live broadcast; and a second layout identifier sending module configured to send the second layout identifier to a client for displaying a group of media streams from a group of users of the live broadcast.
[0100] In some embodiments, the template acquisition request is a first template acquisition request, and the target configuration template file is a first configuration template file. The apparatus 1000 further includes: a second acquisition request receiving module configured to receive a second template acquisition request from a client in response to a second layout identifier not being included in a set of configuration template files and a first configuration template file, the second template acquisition request including the second layout identifier; a second selection module configured to select a second configuration template file corresponding to the second layout identifier from a plurality of configuration template files; and a second configuration template file sending module configured to send the second configuration template file including the second layout identifier to the client to adjust the display of a set of media streams for a set of users in a live broadcast using the second layout corresponding to the second layout identifier.
[0101] In some embodiments, the apparatus 1000 further includes: an acquisition request receiving module configured to receive an acquisition request from the client for acquiring a parsing engine in response to the client not having a configuration template file for the layout of the live media stream; and a fourth configuration template file sending module configured to send the parsing engine and a fourth configuration template file including a layout identifier to the client to acquire a target layout corresponding to the layout identifier using the parsing engine.
[0102] In some embodiments, the template configuration file includes a layout style, a maximum number of users participating in the live chat, and the display position of the video stream for each user.
[0103] Figure 11 A schematic block diagram of an example device 1100 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device client 104, broadcaster 102, and server 106 can be implemented using device 1100. As shown, device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1102 or loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 can also store various programs and data required for the operation of device 1100. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.
[0104] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage page 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0105] The various processes and handling described above, such as methods 200, 300, and 400, can be executed by processing unit 1101. For example, in some embodiments, methods 200, 300, and 400 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU 1101, one or more actions of the example methods 200, 300, and 400 described above can be performed.
[0106] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.
[0107] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0108] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0109] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0110] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0111] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0112] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0114] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A method for handling the layout of a media stream, comprising: The client sends a request associated with the live stream to the server to receive a layout identifier from the server corresponding to a target layout for a set of media streams for a group of users in the live stream. In response to the existence of a set of configuration template files for the layout of the media stream for the live broadcast on the client, it is determined whether the target layout is included in the set of configuration template files based on the layout identifier; In response to the fact that the target layout is not included in the set of configuration template files, a template acquisition request is sent to the server to receive a target configuration template file including the layout identifier from the server, the template acquisition request including the layout identifier; as well as Based on the target configuration template file, the target layout corresponding to the layout identifier is used to display the group of media streams of the group of users in the live broadcast.
2. The method according to claim 1, wherein the layout identifier is determined based on the layout style received from the broadcaster's terminal of the live stream and the maximum number of users participating in the live stream conversation.
3. The method according to claim 2, wherein the layout identifier is a first layout identifier, the target layout is a first layout, and the method further comprises: In response to the target layout being adjusted by the broadcaster during the live broadcast, a second layout identifier corresponding to the adjusted target layout is received; In response to the second layout identifier being included in the set of configuration template files and the target configuration template file, a second layout corresponding to the second layout identifier is obtained from the set of configuration template files and the target configuration template file; as well as Use the second layout to adjust the display of the group of media streams for the group of users in the live stream.
4. The method according to claim 3, wherein the template acquisition request is a first template acquisition request, and the target configuration template file is a first configuration template file, the method further comprising: In response to the fact that the second layout identifier is not included in the set of configuration template files and the first configuration template file, a second template acquisition request is sent to the server to receive a second configuration template file including the second layout identifier from the server. The second template acquisition request includes the second layout identifier. as well as The display of the group of media streams for the group of users in the live stream is adjusted based on the second layout in the second configuration template file.
5. The method according to claim 3, further comprising: In response to the layout identifier being included in a third configuration template file in the set of configuration template files, the target layout corresponding to the layout identifier is obtained from the third configuration template file; as well as The target layout is used to display the group of media streams of the group of users in the live stream.
6. The method of claim 1, wherein displaying the group of media streams of the group of users in the live stream using the target layout corresponding to the layout identifier comprises: The target configuration template file is parsed using a local parsing engine; as well as The target layout corresponding to the layout identifier in the parsed target configuration template file is used to display the group of media streams of the group of users in the live broadcast.
7. The method according to claim 1, wherein determining whether the target layout is included in the set of configuration template files based on the layout identifier comprises: Determine whether the layout identifier exists in the set of configuration template files; as well as In response to the absence of the layout identifier among the plurality of layout identifiers, it is determined that the target layout is not included in the set of configuration template files.
8. The method according to claim 1, further comprising: In response to the fact that the client does not have a configuration template file for the layout of the media stream for the live broadcast, it is determined that the client does not have a parsing engine that can be used to parse the configuration template file; Send a request to the server to obtain the parsing engine; Receive the parsing engine and the fourth configuration template file including the layout identifier from the server; as well as The parsing engine is used to parse the fourth configuration template file to obtain the target layout corresponding to the layout identifier.
9. The method of claim 1, wherein the template configuration file includes a layout style, a maximum number of users participating in the live stream conversation, and a display position of the video stream for each user.
10. A method for handling the layout of a media stream, comprising: In response to receiving a request associated with a live stream from the client at the server, a layout identifier corresponding to the target layout of a set of media streams for a set of users in the live stream is sent to the client; In response to the fact that the target layout corresponding to the layout identifier is not included in a set of configuration template files for the media stream of the live broadcast in the client, a template retrieval request is received from the client, the template retrieval request including the layout identifier; Search among multiple configuration template files for the target configuration template file that includes the layout identifier; as well as The target configuration template file is sent to the client for displaying the group of media streams of the group of users in the live stream.
11. The method of claim 10, further comprising: The maximum number of users who can receive the layout style and participate in the live stream conversation from the broadcaster's end. as well as The layout identifier is determined based on the layout style and the maximum number.
12. The method of claim 11, wherein the layout identifier is a first layout identifier, the target layout is a first layout, and the method further comprises: In response to receiving at least one of an adjusted layout style and a maximum number of adjustments from the broadcaster during the live broadcast, a second layout identifier corresponding to the adjusted target layout is determined. as well as The second layout identifier is sent to the client for displaying the group of media streams of the group of users of the live stream.
13. The method of claim 12, wherein the template acquisition request is a first template acquisition request, and the target configuration template file is a first configuration template file, and the method further comprises: In response to the fact that the second layout identifier is not included in the set of configuration template files and the first configuration template file, a second template retrieval request is received from the client, the second template retrieval request including the second layout identifier; Select the second configuration template file corresponding to the second layout identifier from the plurality of configuration template files; as well as Send a second configuration template file, including the second layout identifier, to the client to adjust the display of the group of media streams for the group of users in the live broadcast using the second layout corresponding to the second layout identifier.
14. The method of claim 10, further comprising: In response to the client not having a configuration template file for the layout of the media stream for the live broadcast, a request for obtaining the parsing engine is received from the client; as well as The parsing engine and a fourth configuration template file including the layout identifier are sent to the client to obtain the target layout corresponding to the layout identifier using the parsing engine.
15. The method of claim 10, wherein the template configuration file includes a layout style, a maximum number of users participating in the live stream conversation, and a display position of the video stream for each user.
16. An apparatus for processing the layout of a media stream, comprising: The layout identifier receiving module is configured to send a request associated with the live stream from the client to the server in order to receive from the server a layout identifier corresponding to the target layout of a set of media streams for a set of users in the live stream. The template matching module is configured to respond to a set of configuration template files for a layout of a media stream for live streaming that exist on the client side. Based on the layout identifier, it determines whether a set of configuration template files includes the target layout. The template retrieval module is configured to send a template retrieval request to the server in response to the target layout not being included in a set of configuration template files, in order to receive the target configuration template file including the layout identifier from the server. The template retrieval request includes the layout identifier. as well as The layout display module is configured to display a set of media streams from a group of users in a live stream, based on a target configuration template file and using a target layout corresponding to the layout identifier.
17. An apparatus for processing the layout of a media stream, comprising: The layout identifier sending module is configured to, in response to receiving a request associated with a live stream from the client at the server, send a layout identifier corresponding to a target layout of a set of media streams for a set of users in the live stream to the client. The template retrieval request receiving module is configured to receive a template retrieval request from the client in response to the fact that the target layout corresponding to the layout identifier is not included in a set of configuration template files for the media stream of the live broadcast in the client. The template retrieval request includes the layout identifier. The search module is configured to search for a target configuration template file that includes the layout identifier from multiple configuration template files; as well as The configuration template file sending module is configured to send the target configuration template file to the client for displaying the group of media streams of the group of users in the live broadcast.
18. An electronic device comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-15.
19. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-15 when executed by a processor.
20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-15.