Updating method of video multi-picture layout, electronic equipment and readable storage medium

Abstract

The invention discloses a video multi-picture layout updating method, electronic equipment and a readable storage medium. The method comprises the steps that a screen mixing request initiated by a terminal is received, and the screen mixing request is used for indicating that the terminal has a preset change; the preset change comprises that the terminal issues or stops issuing a target audio and video stream; on the basis of the screen mixing request, screen mixing information is constructed according to a picture layout rule, and the screen mixing information comprises information for mixing a sub-picture corresponding to the target audio and video stream with a current video multi-picture layout; and updating the video multi-picture layout based on the screen mixing information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and in particular to a method for updating a multi-screen layout of a video, an electronic device, and a readable storage medium. Background Art

[0002] In the context of rapid network development, live video broadcasting is increasingly used in our daily lives. In order to improve the audio / video real-time streaming experience, screen mixing technology has emerged. The existing real-time clock (RTC)-real-time audio and video screen mixing technology requires manual setting of the screen layout. When the number of placeholder anchors changes, the video screen layout cannot be automatically adjusted for screen mixing, which affects the user experience to a certain extent. Summary of the invention

[0003] The embodiments of the present application provide a method for updating a multi-screen layout of a video, an electronic device, and a readable storage medium, which can solve the problem that the existing mixed screen technology cannot automatically adjust the video screen layout.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, a method for updating a video multi-screen layout is provided, the method comprising: receiving a mixed screen request initiated by a terminal, wherein the mixed screen request is used to instruct the terminal to undergo a preset change; the preset change comprises the terminal publishing or stopping publishing a target audio and video stream; based on the mixed screen request, constructing mixed screen information according to a screen layout rule, wherein the mixed screen information comprises information for mixing a sub-screen corresponding to the target audio and video stream with the current video multi-screen layout; and updating the video multi-screen layout based on the mixed screen information.

[0006] In a second aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method for updating the multi-screen layout of the video as described in the first aspect are implemented.

[0007] According to a third aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for updating the multi-screen layout of a video as described in the first aspect are implemented.

[0008] The technical solution provided by this application may have the following beneficial effects:

[0009] In an embodiment of the present application, a mixing screen request initiated by a receiving terminal is used to obtain information about whether the terminal has published or stopped publishing a target audio and video stream. Based on the received mixing screen request, mixing screen information is constructed according to a screen layout rule, wherein the mixing screen information includes information about mixing a sub-screen corresponding to the target audio and video stream with a current video multi-screen layout, thereby updating the video multi-screen layout based on the mixing screen information, and mixing the sub-screen corresponding to the target audio and video stream published or stopped by the terminal with the current video multi-screen layout, so as to automatically update the video multi-screen layout according to the mixing screen information.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 A schematic flow chart of a method for updating a video multi-screen layout provided by an exemplary embodiment of the present application is shown;

[0013] Figure 2 A schematic structural diagram of a system for updating a video multi-screen layout provided by an exemplary embodiment of the present application is shown;

[0014] Figure 3 A schematic flow chart of another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application is shown;

[0015] Figure 4 A schematic flow chart of another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application is shown;

[0016] Figure 5 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0018] Figure 1A schematic diagram showing a flow chart of a method for updating a video multi-screen layout provided by an exemplary embodiment of the present application is shown. Figure 1 , the method comprises the following steps.

[0019] Step 102, receiving a mixed screen request initiated by a terminal;

[0020] The mixed screen request is used to instruct the terminal to undergo a preset change; the preset change includes the terminal publishing or stopping publishing a target audio and video stream.

[0021] Step 104: constructing mixed screen information according to the screen layout rule based on the mixed screen request, wherein the mixed screen information includes information for mixing the sub-screen corresponding to the target audio and video stream with the current video multi-screen layout.

[0022] The screen layout rule may include closing one or more sub-screens existing in the video multi-screen layout; may include adding one or more sub-screens in the video multi-screen layout; and may also include adjusting each sub-screen in the video multi-screen layout, such as adjusting the order of the sub-screens, adjusting the position of the sub-screens, etc. During use, it is necessary to construct the mixed screen information according to the screen layout rule corresponding to the mixed screen request according to the information indicated in the mixed screen request. The mixed screen information indicates the processing of the sub-screen corresponding to the target audio and video stream and the video multi-screen layout, so that the video multi-screen layout can be updated according to the mixed screen information.

[0023] Step 106: Update the video multi-screen layout based on the mixed screen information.

[0024] In an embodiment of the present application, after a preset change occurs in the terminal and a mixing screen request is initiated, the mixing screen request initiated by the terminal is received to obtain information about whether the terminal has published or stopped publishing a target audio and video stream. Based on the received mixing screen request, mixing screen information is constructed according to the screen layout rules, wherein the screen layout rules include a variety of rules for adjusting the video multi-screen layout. At this time, the changes that have occurred in the terminal are determined based on the information in the mixing screen request, and then the rules for adjusting the video multi-screen layout are determined from the screen layout rules based on the changes. This rule is used as the mixing screen information to indicate that the sub-screen corresponding to the target audio and video stream is mixed with the current video multi-screen layout, thereby updating the video multi-screen layout based on the mixing screen information, and mixing the sub-screen corresponding to the target audio and video stream that the terminal has published or stopped publishing with the current video multi-screen layout, so as to automatically update the video multi-screen layout according to the mixing screen information.

[0025] For example, the current terminal publishes a video stream and issues a mixed screen request. By receiving the mixed screen request, it can be learned that the terminal needs to perform mixed screen when publishing the video stream. According to the screen layout rule, the mixed screen information can be determined to add the sub-screen corresponding to the video stream to the current video multi-screen layout, thereby adding the sub-screen corresponding to the video stream to the video multi-screen layout according to the mixed screen information, thereby realizing the update of the video multi-screen layout.

[0026] In one implementation, the above screen layout rule may include a rule for adding a sub-screen or a rule for reducing a sub-screen.

[0027] The rule for adding a sub-picture includes one of the following:

[0028] (1) Displaying the sub-picture corresponding to the target audio and video stream at a preset position of the video multi-picture layout.

[0029] Optionally, the mixed screen request initiated by the received terminal may carry information that the sub-screen corresponding to the target audio and video stream is displayed at the preset position of the video multi-screen layout, and the sub-screen corresponding to the target audio and video stream can be displayed at the preset position of the video multi-screen layout according to the information; or the sub-screens corresponding to the newly added target audio and video streams may be preset to be displayed at the preset position of the video multi-screen layout, and the sub-screen corresponding to the target audio and video stream may be displayed at the preset position of the video multi-screen layout according to the current preset.

[0030] (2) Displaying the sub-picture corresponding to the target audio and video stream in the first free position of the video multi-picture layout.

[0031] Among them, when there are multiple free positions in the current video multi-screen layout and it is not specified to add the sub-screen corresponding to the target audio and video stream to the preset position, considering the coordination of the screen, the sub-screen corresponding to the newly added target audio and video stream is displayed in the first free position of the video multi-screen layout.

[0032] Optionally, the first free position in the above-mentioned video multi-screen layout can be the first free position horizontally adjacent to the sub-screen already existing in the current video multi-screen layout, or can be the first free position vertically adjacent to the sub-screen already existing in the current video multi-screen layout.

[0033] (3) When there is no free space in the video multi-screen layout, increase the number of screens in the video multi-screen layout.

[0034] For example, if there are sub-screens in all six positions of the current video multi-screen layout and there are no empty positions, then when a sub-screen corresponding to the target audio and video stream needs to be added, the number of screens in the video multi-screen layout is increased to 9, and then the sub-screen corresponding to the target audio and video stream can be displayed in the video multi-screen layout according to the above rules (1) or (2).

[0035] (4) When the number of pictures in the video multi-picture layout reaches a first threshold, the sub-picture corresponding to the target audio and video stream is not displayed, where the first threshold is the highest level of the number of pictures.

[0036] Among them, when the number of pictures in the current video multi-screen layout has reached the highest level of pictures, for example, the number of pictures in the preset video multi-screen layout includes three levels, namely 3, 6, and 9, and the current video multi-screen layout is at the highest level of 9 pictures, and sub-pictures already exist at the positions of the 9 pictures, then the sub-picture corresponding to the newly added target audio and video stream will not be mixed, and the sub-picture corresponding to the target audio and video stream will not be displayed in the video multi-screen layout.

[0037] The sub-picture reduction rule includes at least one of the following:

[0038] (1) removing the sub-picture corresponding to the target audio and video stream, and moving the sub-picture that already exists in the video multi-picture layout forward;

[0039] Among them, for the change of stopping publishing the target audio and video stream in the mixed screen request initiated by the terminal, the sub-screen corresponding to the target audio and video stream can be removed from the video multi-screen layout, and the sub-screen in the existing video multi-screen layout can be moved forward, wherein the subsequent multiple sub-screens can be moved forward in sequence, or one of the subsequent multiple sub-screens can be moved forward to the position where the sub-screen corresponding to the above-mentioned target audio and video stream is removed, filling the gap caused by removing the sub-screen corresponding to the above-mentioned target audio and video stream, so that the updated video multi-screen layout is more coordinated.

[0040] In one implementation, after removing the sub-picture corresponding to the target audio and video stream, the method further includes: reducing the number of pictures in the video multi-picture layout when the number of pictures in the video multi-picture layout does not exceed the second threshold; wherein the second threshold is less than the number of pictures at the highest level.

[0041] For example, the number of pictures in the video multi-screen layout includes three levels, namely 3, 6, and 9. The current number of pictures in the video multi-screen layout is 9, and the number of sub-screens corresponding to the existing audio and video streams is 7. After removing the sub-screen corresponding to a target audio and video stream, the number of occupied pictures in the video multi-screen layout is 6, which is equal to the second threshold. Then the number of pictures in the video multi-screen layout can be reduced to 6.

[0042] (2) Display the sub-screen corresponding to the audio and video stream to be mixed in the first free position of the video multi-screen layout.

[0043] Among them, the audio and video stream to be mixed can be the target audio and video stream that was previously requested by the terminal to be mixed, but the mixing process was not performed because there was no free position in the video multi-screen layout. At this time, when there is a free position in the video multi-screen layout, the sub-screen corresponding to the target audio and video stream can be displayed in the first free position of the video multi-screen layout. Optionally, the first free position of the video multi-screen layout can be the first free position adjacent to the sub-screen that already exists in the current video multi-screen layout in the horizontal direction, or it can be the first free position adjacent to the sub-screen that already exists in the current video multi-screen layout in the vertical direction.

[0044] In one implementation, the above step 104 may construct mixed screen information based on the mixed screen request and according to the screen layout rule, which may include one of the following:

[0045] (1) When the terminal publishes the target audio stream, the mixed screen information is constructed according to the rule for adding sub-pictures.

[0046] In an embodiment of the present application, a mixing screen request initiated by a receiving terminal indicates that the terminal has released a target audio stream and has allowed the audio stream to participate in the mixing screen. The mixing screen information is constructed according to the rule of adding sub-screens. For example, the mixing screen information may be to display the sub-screen corresponding to the target audio stream at a preset position of the video multi-screen layout, or to display the sub-screen corresponding to the target audio stream at the first free position of the video multi-screen layout, or to increase the number of screens of the video multi-screen layout when there is no free position in the video multi-screen layout, and then display the sub-screen corresponding to the target audio stream at the preset position or the first free position, so that the video multi-screen layout can be updated according to the mixing screen information.

[0047] (2) When the terminal stops publishing the target audio stream, the mixed screen information is constructed according to the sub-picture reduction rule.

[0048] In an embodiment of the present application, by receiving a mixed screen request initiated by a terminal to indicate to stop publishing a target audio stream, it is learned that the terminal has previously published the target audio stream but is currently stopping publishing the target audio stream, and mixed screen information is constructed according to a rule for reducing sub-screens. The mixed screen information may include removing the sub-screen corresponding to the target audio stream that has been published by the terminal from the video multi-screen layout, and leaving the position occupied by the sub-screen corresponding to the target audio stream in an idle state so that it can be occupied by other audio and video streams; and the mixed screen information may also include moving the sub-screen that already exists in the video multi-screen layout forward to fill the vacancy caused by removing the sub-screen corresponding to the target audio stream, thereby implementing an update of the video multi-screen layout according to the mixed screen information.

[0049] In one implementation, the above step 104 may construct mixed screen information based on the mixed screen request and according to the screen layout rule, which may include one of the following:

[0050] (1) when the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures;

[0051] In an embodiment of the present application, a mixed screen request indicating the release of a target video stream is initiated by a receiving terminal, and mixed screen information is constructed according to a rule for adding sub-screens. For example, when the position in the video multi-screen layout where the sub-screen corresponding to the target video stream released by the terminal is displayed is specified, the mixed screen information includes displaying the sub-screen corresponding to the target video stream at the specified position of the video multi-screen layout; and when the position is not specified and there is an idle position in the current video multi-screen layout, the mixed screen information may include displaying the sub-screen corresponding to the target video stream at the first idle position in the video multi-screen layout, thereby mixing the sub-screen corresponding to the target video stream with the video multi-screen layout according to the mixed screen information to update the video multi-screen layout.

[0052] In one implementation, when the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures may include: when the terminal publishes the target video stream and the terminal has published a historical audio stream, the constructed mixed screen information includes replacing the sub-picture corresponding to the historical audio stream with the sub-picture corresponding to the target video stream.

[0053] (2) When the terminal stops publishing the target video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

[0054] In an embodiment of the present application, a mixed screen request instructing to stop publishing a target video stream is initiated by a receiving terminal, and mixed screen information is constructed according to a rule for reducing sub-screens. For example, the mixed screen information may include removing the sub-screen corresponding to the target video stream that has been published by the terminal from the video multi-screen layout, and leaving the position occupied by the sub-screen corresponding to the target video stream in an idle state so that it can be occupied by other audio and video streams; and the mixed screen information may also include moving the sub-screen that already exists in the video multi-screen layout forward to fill the vacancy caused by removing the sub-screen corresponding to the target video stream, thereby updating the video multi-screen layout according to the mixed screen information.

[0055] In an implementation manner, when the terminal stops publishing the target video stream, constructing the mixed screen information according to the sub-picture reduction rule may include one of the following:

[0056] (a) when the terminal stops publishing the target video stream and the terminal has published the historical audio stream, replacing the sub-picture corresponding to the target video stream with the sub-picture corresponding to the historical audio stream;

[0057] (b) When the terminal stops publishing the target video stream and has published the historical video stream, the sub-picture corresponding to the target video stream is replaced with the sub-picture corresponding to the historical video stream.

[0058] In the embodiment of the present application, there are two situations when the terminal stops publishing the target video stream. One is that the terminal has published a historical audio stream, which is an audio stream other than the target video stream published by the terminal before the terminal stops publishing the target video stream. Then the constructed mixed screen information includes replacing the sub-picture corresponding to the target video stream with the sub-picture corresponding to the historical audio stream; the other is that the terminal has published a historical video stream, which is a video stream other than the target video stream published by the terminal before the terminal stops publishing the target video stream. Then the constructed mixed screen information includes replacing the sub-picture corresponding to the target video stream with the sub-picture corresponding to the historical video stream. In this way, the sub-picture corresponding to the target video stream is removed, and the vacancy caused by removing the sub-picture corresponding to the target video stream is filled, thereby realizing the update of the multi-picture layout of the video.

[0059] In one implementation, the above-mentioned mixed screen request is also used to indicate exiting mixed screen. Step 104 constructs mixed screen information according to the screen layout rule based on the mixed screen request, which may include: removing all sub-screens in the current video multi-screen layout; or constructing the mixed screen information according to the sub-screen reduction rule.

[0060] In an embodiment of the present application, when a screen mixing request is received from a terminal indicating exiting screen mixing, the screen mixing information may include removing all existing sub-screens in the current video multi-screen layout, at which time the terminal ends the video live broadcast. It is also possible to choose to construct the screen mixing information according to the sub-screen reduction rule, and the screen mixing information may include removing the sub-screen corresponding to the target audio and video stream of the terminal requesting to exit the screen mixing, and moving forward the sub-screen corresponding to the target audio and video stream published by other terminals that already exist in the video multi-screen layout; and after removing the sub-screen corresponding to the target audio and video stream of the terminal requesting to exit the screen mixing, the sub-screen corresponding to the audio and video stream to be mixed can be displayed in the vacant position due to the removal of the sub-screen corresponding to the target audio and video stream, so as to perform corresponding screen mixing processing according to the constructed screen mixing information and update the video multi-screen layout.

[0061] In one implementation, the above step 104 may construct mixed screen information based on the mixed screen request and according to the screen layout rule, which may include one of the following:

[0062] (1) when the terminal publishes an auxiliary stream of the target audio and video stream and specifies that a sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, constructing the mixed screen information according to the sub-picture adding rule;

[0063] Among them, when the terminal publishes an auxiliary stream of the target audio and video stream and specifies that the sub-screen corresponding to the auxiliary stream is located at a preset position of the video multi-screen layout, the mixed screen information constructed according to the sub-screen addition rule may include displaying the sub-screen corresponding to the auxiliary stream at the preset position of the video multi-screen layout.

[0064] (2) when the terminal publishes an auxiliary stream of the target audio and video stream and does not specify that the sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, the sub-picture corresponding to the auxiliary stream is not displayed;

[0065] Among them, when the terminal publishes an auxiliary stream of the target audio and video stream and does not specify that the sub-screen corresponding to the auxiliary stream is located at a preset position of the video multi-screen layout, the constructed mixed screen information may include not displaying the sub-screen corresponding to the auxiliary stream.

[0066] (3) When the terminal stops publishing the auxiliary stream of the target audio and video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

[0067] Among them, when the terminal stops publishing the auxiliary stream of the target audio and video stream, the mixed screen information constructed according to the sub-screen reduction rule may include removing the sub-screen corresponding to the auxiliary stream, moving the existing sub-screen in the video multi-screen layout forward, or displaying the sub-screen corresponding to the audio and video stream to be mixed screen in the position that is vacant after the sub-screen corresponding to the auxiliary stream is removed. Thus, according to the mixed screen request initiated by the terminal, the corresponding mixed screen information is constructed in different situations, and the video multi-screen layout can be updated according to the mixed screen information.

[0068] In one implementation, the above-mentioned screen layout rule may further include: when the terminal publishes multiple target audio and video streams, retaining a sub-screen corresponding to the target audio and video stream in the video multi-screen layout.

[0069] Among them, by limiting the number of sub-screens corresponding to the target audio and video streams released by the display terminal in the video multi-screen layout, the attention to a single target audio and video stream is guaranteed, avoiding a single terminal from occupying too much of the position of the sub-screen in the video multi-screen layout, and enabling multiple terminals to participate.

[0070] In one implementation, after receiving the screen mixing request initiated by the terminal in the above step 102, the method may further include: screening the historical audio and video streams released by the terminal.

[0071] In an embodiment of the present application, after receiving a mixing screen request initiated by a terminal, for the historical audio stream published by the terminal, it is determined whether the historical audio stream participates in the current mixing screen and whether to retain or discard it; and if the historical audio and video stream is not specified to be displayed at a preset position in the video multi-screen layout and the historical audio and video stream is an auxiliary stream, the historical audio and video stream is discarded. By the above-mentioned screening of the historical audio and video streams published by the terminal, unnecessary occupation of the video multi-screen layout by redundant historical audio and video streams is avoided.

[0072] Figure 2A structural schematic diagram of a video multi-screen layout update system provided by an exemplary embodiment of the present application is shown, the system comprising: a control center and a media processing center; wherein the control center accesses the terminal through a cloud access gateway module 21, the control center comprises an access scheduling module 22, a stream access control module 23, a room control module 24, a stream management module 25, a message master control module 26, a message distribution module 27, and a region management service (Region Manager Service, RMS) module 28; the media processing center comprises a stream access negotiation module 29, a media processing service (Media Processing Center, MPC) module 210, a media forwarding (Selective Forwarding Unit, SFU) module 211, a video transcoding synthesis (Video Processor Server, VPS) module 212, a streaming media service (Streaming media Service, SS) module 213, a media management module 214, and a content delivery network (Content Delivery Network, CDN) module 215.

[0073] Among them, the access scheduling module 22 is used to provide access point allocation services; the stream access control module 23 is used to provide the opening of stream-related capabilities, including stream publishing and subscription, etc., mainly used for non-room models; the room control module 24 is used to provide the opening of room-related capabilities, including joining a room, exiting a room, publishing streams in a room, subscribing to streams, etc.; the stream management module 25 is used to publish and subscribe to internal streams of the real-time audio and video transmission network (RTN), including cascade scheduling of streams; the message master control module 26 is used to provide the ability to allocate message distribution addresses; the message distribution module 27 is used to provide message subscription and publishing, and cooperates with the message master control module 26; the regional management service module 28 is used to provide edge distribution services; the stream access negotiation module 29 is used to provide media negotiation capabilities and link detection capabilities; the media processing service module 210 is used to provide media processing agent services; the media forwarding module 211 is used to provide media forwarding services; the video transcoding synthesis module 212 is used to provide video transcoding synthesis services; the streaming media service module 213 is used to provide recording and streaming; the media control module 214 is used to provide control services for the media server.

[0074] Figure 3 A schematic diagram showing a flow chart of another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application is shown. Figure 3 , the method comprises the following steps.

[0075] Step 301: The terminal initiates a mixed screen request;

[0076] Step 302, the control center reads the published streams in the room and sorts and stores them in the database;

[0077] The published flows can be sorted by various rules, such as the flow size, the time when the flow is published, etc.

[0078] Step 303, filter the published streams;

[0079] Among them, for the audio stream in the published stream, it is determined whether to retain or discard it according to whether the audio stream participates in screen mixing; if the published stream is an auxiliary stream and no preset position is specified, the stream is discarded.

[0080] Step 304, storing the target streams in the screen mixing request sent by the terminal in order in the list to be mixed;

[0081] The published flows can be sorted by various rules, such as the flow size, the time when the flow is published, etc.

[0082] Step 305, selecting a target stream with the maximum number of pictures that can be displayed in the current video multi-picture layout from the list of screens to be mixed;

[0083] For example, the maximum number of screens that can be displayed in the current video multi-screen layout is 9, and 9 target streams are selected from the list of screens to be mixed.

[0084] Step 306, determining a corresponding number of target streams according to the number of positions of idle sub-pictures in the current video multi-picture layout;

[0085] Step 307, constructing mixed screen information according to the screen layout rule and storing it in a database;

[0086] Step 308, sending the constructed mixed screen information to the media processing center;

[0087] Step 309: The media processing center mixes the target stream published by the terminal according to the mixing information, and updates the video multi-screen layout.

[0088] In an embodiment of the present application, after the terminal initiates a mixed screen request, the control center first filters the published streams in the room, then constructs mixed screen information for the target stream and informs the media control center, so that the media control center can mix the target stream published by the terminal according to the mixed screen information, thereby realizing the update of the video multi-screen layout.

[0089] Figure 4 A schematic diagram showing a flow chart of another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application is shown. Figure 4 , the method comprises the following steps.

[0090] Step 401: The terminal initiates a mixed screen request, wherein the mixed screen request is used to instruct the terminal to exit the room;

[0091] Step 402, the control center reads the stored mixed screen information;

[0092] Step 403, judging whether the terminal exiting the room is the main initiator of screen mixing according to the stored screen mixing information, if so, notifying the media control center to stop screen mixing; if not, constructing screen mixing information according to the sub-screen reduction rule in the screen layout rule;

[0093] Step 404, sending the constructed mixed screen information to the media processing center;

[0094] Step 405: The media processing center removes the target video stream published by the terminal according to the mixed screen information, and updates the video multi-screen layout.

[0095] In an embodiment of the present application, when a terminal requests to exit a room, it is necessary to determine whether the terminal is the main initiator of screen mixing. If so, the control center notifies the media control center to stop screen mixing, and the room no longer performs screen mixing. If not, the control center constructs screen mixing information according to the sub-screen reduction rule in the screen layout rule, so that the media control center removes the target video stream published by the terminal according to the mixed screen information, thereby updating the video multi-screen layout. At this time, other terminals can still perform screen mixing in this room, which improves the experience of updating the video multi-screen layout.

[0096] In an exemplary embodiment, an electronic device is also provided, the electronic device is used to execute the above-mentioned video multi-screen layout update method, Figure 5 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call a computer program stored in the memory 503 and can be run on the processor 501 to execute all or part of the steps of the above-mentioned video multi-screen layout update method embodiment, and can achieve the same technical effect, to avoid repetition, it will not be repeated here.

[0097] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0098] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0099] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0100] In an exemplary embodiment, a readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement all or part of the steps in the above-mentioned method for updating the multi-screen layout of the video. For example, the readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0101] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0102] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for updating a video multi-screen layout, characterized in that: include: Receiving a mixed screen request initiated by a terminal, wherein the mixed screen request is used to instruct the terminal to undergo a preset change; the preset change includes the terminal publishing or stopping publishing a target audio and video stream; Based on the screen mixing request, constructing screen mixing information according to the screen layout rule, wherein the screen mixing information includes information for mixing the sub-screen corresponding to the target audio and video stream with the current video multi-screen layout; The video multi-screen layout is updated based on the mixed screen information.

2. The method according to claim 1, characterized in that The screen layout rule includes a rule for adding sub-screens or a rule for reducing sub-screens; wherein, The rule for adding a sub-picture includes one of the following: Displaying the sub-picture corresponding to the target audio and video stream at a preset position of the video multi-picture layout; Displaying the sub-picture corresponding to the target audio and video stream in the first free position of the video multi-picture layout; In the case where there is no free space in the video multi-screen layout, increasing the number of screens in the video multi-screen layout; When the number of pictures in the video multi-picture layout reaches a first threshold, the sub-picture corresponding to the target audio and video stream is not displayed; wherein the first threshold is the number of pictures at the highest level; The sub-picture reduction rule includes at least one of the following: Removing the sub-picture corresponding to the target audio and video stream, and moving the sub-picture that already exists in the video multi-picture layout forward; The sub-screen corresponding to the audio and video stream to be mixed is displayed in the first free position of the video multi-screen layout.

3. The method according to claim 2, characterized in that After removing the sub-picture corresponding to the target audio and video stream, the method further includes: When the number of pictures in the video multi-picture layout does not exceed a second threshold, reducing the number of pictures in the video multi-picture layout; The second threshold is a number of pictures less than the highest level.

4. The method according to claim 2, characterized in that: The constructing the mixed screen information according to the screen layout rule based on the mixed screen request includes one of the following: In the case where the terminal publishes the target audio stream, constructing the mixed screen information according to the rule of adding sub-pictures; In a case where the terminal stops publishing the target audio stream, the mixed screen information is constructed according to the sub-picture reduction rule.

5. The method according to claim 2, characterized in that: The constructing the mixed screen information according to the screen layout rule based on the mixed screen request includes one of the following: In the case where the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures; In the case where the terminal stops publishing the target video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

6. The method according to claim 5, characterized in that When the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures includes: In the case that the terminal publishes the target video stream and has published the historical audio stream, the sub-picture corresponding to the historical audio stream is replaced with the sub-picture corresponding to the target video stream.

7. The method according to claim 5, characterized in that The step of constructing the mixed screen information according to the sub-picture reduction rule when the terminal stops publishing the target video stream includes one of the following: When the terminal stops publishing the target video stream and has published the historical audio stream, replacing the sub-picture corresponding to the target video stream with the sub-picture corresponding to the historical audio stream; When the terminal stops publishing the target video stream and has published the historical video stream, the sub-picture corresponding to the target video stream is replaced with the sub-picture corresponding to the historical video stream.

8. The method according to claim 2, characterized in that: The mixed screen request is also used to indicate exiting the mixed screen; The constructing mixed screen information based on the mixed screen request and according to the screen layout rule includes: Remove all sub-screens from the current video multi-screen layout; or, The mixed screen information is constructed according to the sub-picture reduction rule.

9. The method according to claim 2, characterized in that: The constructing the mixed screen information according to the screen layout rule based on the mixed screen request includes one of the following: When the terminal publishes the auxiliary stream of the target audio and video stream and specifies that the sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, constructing the mixed screen information according to the sub-picture adding rule; When the terminal publishes an auxiliary stream of the target audio and video stream and does not specify that the sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, the sub-picture corresponding to the auxiliary stream is not displayed; In a case where the terminal stops publishing the auxiliary stream of the target audio and video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

10. The method according to claim 1, characterized in that The screen layout rules also include: In the case where the terminal publishes multiple target audio and video streams, a sub-picture corresponding to the target audio and video stream is reserved in the video multi-picture layout.

11. The method according to claim 1, characterized in that: After the mixed screen request initiated by the receiving terminal, the method further includes: Filter the historical audio and video streams published by the terminal.

12. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for updating the video multi-screen layout as described in any one of claims 1 to 11 are implemented.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for updating the video multi-screen layout as described in any one of claims 1 to 11 are implemented.

Images