Video multi-picture layout updating method, electronic device, and readable storage medium

Abstract

The present application discloses a video multi-picture layout updating method, an electronic device, and a readable storage medium. The method comprises: receiving a screen mixing request initiated by a terminal, wherein the screen mixing request is used for indicating that the terminal has a preset change, and the preset change comprises the terminal publishing or stopping publishing a target audio and video stream; on the basis of the screen mixing request, constructing screen mixing information according to a picture layout rule, wherein the screen mixing information comprises information for mixing a sub-picture corresponding to the target audio and video stream with a current video multi-picture layout; and updating the video multi-picture layout on the basis of the screen mixing information.

Description

Video multi-screen layout updating method, electronic device and readable storage medium

[0001] Cross-references

[0002] This application claims priority to a Chinese patent application filed with the Patent Office of China on August 23, 2023, with application number 202311072303.X and invention name “Method for updating video multi-screen layout, electronic device and readable storage medium”. The entire contents of the application are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of communication technology, and in particular to a method for updating a multi-screen layout of a video, an electronic device, and a readable storage medium. Background Art

[0004] With the rapid development of the internet, live video streaming is becoming increasingly commonplace. To improve the real-time audio and video streaming experience, screen mixing technology has emerged. Existing real-time clock (RTC)-based screen mixing for real-time audio and video requires manual layout configuration. This prevents automatic adjustment of the video layout for screen mixing when the number of live streamers changes, significantly impacting the user experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method for updating a multi-screen layout of a video, an electronic device, and a readable storage medium, which can solve the problem that the existing mixed screen technology cannot automatically adjust the video screen layout.

[0007] In order to solve the above technical problems, this application is implemented as follows:

[0008] In a first aspect, a method for updating a video multi-screen layout is provided, the method comprising: receiving a mixing screen request initiated by a terminal, wherein the mixing screen request is used to instruct the terminal to undergo a preset change; the preset change includes the terminal publishing or stopping publishing a target audio and video stream; based on the mixing screen request, constructing mixing screen information according to a screen layout rule, wherein the mixing screen information includes information for mixing a sub-screen corresponding to the target audio and video stream with the current video multi-screen layout; and updating the video multi-screen layout based on the mixing screen information.

[0009] In a second aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method for updating the multi-screen layout of the video as described in the first aspect are implemented.

[0010] In a third aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for updating the multi-screen layout of a video as described in the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] FIG1 is a schematic flow chart showing a method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application;

[0013] FIG2 shows a schematic structural diagram of a system for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application;

[0014] FIG3 is a schematic flow chart showing another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application;

[0015] FIG4 is a flow chart showing another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application;

[0016] FIG5 shows a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0018] FIG1 shows a flow chart of a method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application. Referring to FIG1 , the method includes the following steps.

[0019] Step 102: Receive a screen mixing request initiated by a terminal.

[0020] The mixed screen request is used to instruct the terminal to undergo a preset change; the preset change includes the terminal publishing or stopping publishing a target audio and video stream.

[0021] Step 104 : constructing screen mixing information based on the screen mixing request and according to screen layout rules, wherein the screen mixing information includes information for mixing the sub-screen corresponding to the target audio and video stream with the current video multi-screen layout.

[0022] The screen layout rules may include closing one or more existing sub-screens in the video multi-screen layout; adding one or more sub-screens to the video multi-screen layout; or adjusting the sub-screens in the video multi-screen layout, such as adjusting the order or position of the sub-screens. During use, it is necessary to construct the mixing information according to the information indicated in the mixing request and the screen layout rules corresponding to the mixing request. The mixing information indicates the processing of the sub-screens corresponding to the target audio and video streams and the video multi-screen layout, so that the video multi-screen layout can be updated according to the mixing information.

[0023] Step 106: Update the video multi-screen layout based on the mixed screen information.

[0024] In an embodiment of the present application, after a preset change occurs in the terminal and a mixing screen request is initiated, the mixing screen request initiated by the terminal is received to obtain information about whether the terminal has published or stopped publishing the target audio and video stream. Based on the received mixing screen request, the mixing screen information is constructed according to the screen layout rules, wherein the screen layout rules include a variety of rules for adjusting the video multi-screen layout. At this time, the changes that have occurred in the terminal are determined based on the information in the mixing screen request, and then the rules for adjusting the video multi-screen layout are determined from the screen layout rules based on the changes. This rule is used as the mixing screen information to indicate that the sub-screen corresponding to the target audio and video stream is mixed with the current video multi-screen layout, thereby updating the video multi-screen layout based on the mixing screen information, and mixing the sub-screen corresponding to the target audio and video stream published or stopped by the terminal with the current video multi-screen layout, so as to automatically update the video multi-screen layout according to the mixing screen information.

[0025] For example, the current terminal publishes a video stream and issues a mixed screen request. By receiving the mixed screen request, it can be known that the video stream published by the terminal needs to be mixed screen. According to the screen layout rules, the mixed screen information can be determined to add the sub-screen corresponding to the video stream to the current video multi-screen layout, thereby adding the sub-screen corresponding to the video stream to the video multi-screen layout according to the mixed screen information to realize the update of the video multi-screen layout.

[0026] In one implementation, the above screen layout rule may include a rule for adding a sub-screen or a rule for reducing a sub-screen.

[0027] The rule for adding a sub-picture includes one of the following:

[0028] (1) Displaying the sub-picture corresponding to the target audio and video stream at a preset position of the video multi-picture layout.

[0029] Optionally, the mixed screen request initiated by the received terminal may carry information to display the sub-screen corresponding to the target audio and video stream at the preset position of the video multi-screen layout, and the sub-screen corresponding to the target audio and video stream may be displayed at the preset position of the video multi-screen layout according to the information; or the sub-screens corresponding to the newly added target audio and video streams may be preset to be displayed at the preset position of the video multi-screen layout, and the sub-screen corresponding to the target audio and video stream may be displayed at the preset position of the video multi-screen layout according to the current preset.

[0030] (2) Displaying the sub-picture corresponding to the target audio and video stream in the first free position of the video multi-picture layout.

[0031] Among them, when there are multiple idle positions in the current video multi-screen layout and it is not specified to add the sub-screen corresponding to the target audio and video stream to the preset position, considering the coordination of the screen, the sub-screen corresponding to the newly added target audio and video stream is displayed in the first idle position of the video multi-screen layout.

[0032] Optionally, the first free position in the above-mentioned video multi-screen layout can be the first free position horizontally adjacent to the sub-screen already existing in the current video multi-screen layout, or the first free position vertically adjacent to the sub-screen already existing in the current video multi-screen layout.

[0033] (3) When there is no free space in the video multi-screen layout, the number of screens in the video multi-screen layout is increased.

[0034] For example, if there are sub-pictures in all six positions of the current video multi-screen layout and there are no vacant positions, when a sub-picture corresponding to the target audio and video stream needs to be added, the number of pictures in the video multi-screen layout is increased to 9, and then the sub-picture corresponding to the target audio and video stream can be displayed in the video multi-screen layout according to the above-mentioned rules (1) or (2).

[0035] (4) When the number of pictures in the video multi-picture layout reaches a first threshold, the sub-picture corresponding to the target audio and video stream is not displayed, where the first threshold is the highest level of picture number.

[0036] Among them, when the number of pictures in the current video multi-screen layout has reached the highest level of pictures, for example, the preset number of pictures in the video multi-screen layout includes three levels, namely 3, 6, and 9, and the video multi-screen layout is currently at the highest level with 9 pictures, and sub-pictures already exist at the positions of the 9 pictures, then the sub-picture corresponding to the newly added target audio and video stream will not be mixed, and the sub-picture corresponding to the target audio and video stream will not be displayed in the video multi-screen layout.

[0037] The sub-picture reduction rule includes at least one of the following:

[0038] (1) The sub-picture corresponding to the target audio and video stream is removed, and the sub-picture existing in the video multi-picture layout is moved forward.

[0039] Among them, for the change of stopping publishing the target audio and video stream in the mixed screen request initiated by the terminal, the sub-screen corresponding to the target audio and video stream can be removed from the video multi-screen layout, and the sub-screen in the existing video multi-screen layout can be moved forward, wherein the subsequent multiple sub-screens can be moved forward in sequence, or one of the subsequent multiple sub-screens can be moved forward to the position where the sub-screen corresponding to the above-mentioned target audio and video stream is removed, filling the gap caused by removing the sub-screen corresponding to the above-mentioned target audio and video stream, so that the updated video multi-screen layout is more coordinated.

[0040] In one implementation, after removing the sub-picture corresponding to the target audio and video stream, the method further includes: reducing the number of pictures in the video multi-picture layout when the number of pictures in the video multi-picture layout does not exceed the second threshold; wherein the second threshold is less than the number of pictures at the highest level.

[0041] For example, the number of pictures in the video multi-screen layout includes three levels, namely 3, 6, and 9. The current number of pictures in the video multi-screen layout is 9, and the number of sub-pictures corresponding to the existing audio and video streams is 7. After removing the sub-picture corresponding to a target audio and video stream, the number of occupied pictures in the video multi-screen layout is 6, which is equal to the second threshold. Then the number of pictures in the video multi-screen layout can be reduced to 6.

[0042] (2) Display the sub-screen corresponding to the audio and video stream to be mixed in the first free position of the video multi-screen layout.

[0043] Among them, the audio and video stream to be mixed can be the target audio and video stream that was previously requested by the terminal to be mixed but was not mixed because there was no free position in the video multi-screen layout. At this time, when there is a free position in the video multi-screen layout, the sub-screen corresponding to the target audio and video stream can be displayed in the first free position in the video multi-screen layout. Optionally, the first free position in the video multi-screen layout can be the first free position horizontally adjacent to the sub-screen that already exists in the current video multi-screen layout, or it can be the first free position vertically adjacent to the sub-screen that already exists in the current video multi-screen layout.

[0044] In one implementation, step 104 of constructing mixed screen information based on the mixed screen request and according to the screen layout rule may include one of the following:

[0045] (1) When the terminal publishes the target audio stream, the mixed screen information is constructed according to the rule for adding sub-pictures.

[0046] In an embodiment of the present application, a mixing screen request is initiated by a receiving terminal, which indicates that the terminal has released a target audio stream and has allowed the audio stream to participate in the mixing screen. The mixing screen information is constructed according to the rule of adding sub-screens. For example, the mixing screen information may be to display the sub-screen corresponding to the target audio stream at a preset position of the video multi-screen layout, or to display the sub-screen corresponding to the target audio stream at the first free position of the video multi-screen layout, or to increase the number of screens of the video multi-screen layout when there is no free position in the video multi-screen layout, and then display the sub-screen corresponding to the target audio stream at the preset position or the first free position, so that the video multi-screen layout can be updated according to the mixing screen information.

[0047] (2) When the terminal stops publishing the target audio stream, the mixed screen information is constructed according to the sub-picture reduction rule.

[0048] In an embodiment of the present application, by receiving a mixing screen request initiated by a terminal to indicate to stop publishing the target audio stream, it is learned that the terminal has previously published the target audio stream but is currently stopping publishing the target audio stream, and mixing screen information is constructed according to the sub-screen reduction rule. The mixing screen information may include removing the sub-screen corresponding to the target audio stream published by the terminal from the video multi-screen layout, and leaving the position occupied by the sub-screen corresponding to the target audio stream in an idle state, which can be occupied by other audio and video streams; and the mixing screen information may also include moving the sub-screen already existing in the video multi-screen layout forward to fill the vacancy caused by removing the sub-screen corresponding to the target audio stream, thereby realizing the update of the video multi-screen layout according to the mixing screen information.

[0049] In one implementation, step 104 of constructing mixed screen information based on the mixed screen request and according to the screen layout rule may include one of the following:

[0050] (1) When the terminal publishes the target video stream, the mixed screen information is constructed according to the rule for adding sub-pictures.

[0051] In an embodiment of the present application, a mixed screen request indicating the release of a target video stream is received by a terminal, and mixed screen information is constructed according to a rule for adding sub-pictures. For example, when the position in the video multi-picture layout at which the sub-picture corresponding to the target video stream released by the terminal is displayed is specified, the mixed screen information includes displaying the sub-picture corresponding to the target video stream at the specified position in the video multi-picture layout; and when the position is not specified and there is an idle position in the current video multi-picture layout, the mixed screen information may include displaying the sub-picture corresponding to the target video stream at the first idle position in the video multi-picture layout, thereby mixing the sub-picture corresponding to the target video stream with the video multi-picture layout according to the mixed screen information to update the video multi-picture layout.

[0052] In one implementation, when the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures may include: when the terminal publishes the target video stream and the terminal has published the historical audio stream, the mixed screen information constructed at this time includes replacing the sub-picture corresponding to the historical audio stream with the sub-picture corresponding to the target video stream.

[0053] (2) When the terminal stops publishing the target video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

[0054] In an embodiment of the present application, a mixed screen request is initiated by a receiving terminal to indicate that the target video stream should be stopped from being published, and mixed screen information is constructed according to the sub-screen reduction rule. For example, the mixed screen information may include removing the sub-screen corresponding to the target video stream published by the terminal from the video multi-screen layout, and leaving the position occupied by the sub-screen corresponding to the target video stream in an idle state, which can be occupied by other audio and video streams; and the mixed screen information may also include moving the sub-screen already existing in the video multi-screen layout forward to fill the vacancy caused by removing the sub-screen corresponding to the target video stream, thereby implementing an update of the video multi-screen layout according to the mixed screen information.

[0055] In one implementation, when the terminal stops publishing the target video stream, constructing the mixed screen information according to the sub-picture reduction rule may include one of the following:

[0056] (a) When the terminal stops publishing the target video stream and has published the historical audio stream, the sub-picture corresponding to the target video stream is replaced with the sub-picture corresponding to the historical audio stream.

[0057] (b) When the terminal stops publishing the target video stream and has published a historical video stream, the sub-picture corresponding to the target video stream is replaced with the sub-picture corresponding to the historical video stream.

[0058] In an embodiment of the present application, when a terminal stops publishing a target video stream, there are two situations: one is that the terminal has published a historical audio stream, which is an audio stream other than the target video stream published by the terminal before the terminal stops publishing the target video stream. In this case, the constructed mixed screen information includes replacing the sub-picture corresponding to the target video stream with the sub-picture corresponding to the historical audio stream; the other is that the terminal has published a historical video stream, which is a video stream other than the target video stream published by the terminal before the terminal stops publishing the target video stream. In this case, the sub-picture corresponding to the target video stream is removed, and the vacancy caused by the removal of the sub-picture corresponding to the target video stream is filled, thereby updating the multi-picture layout of the video.

[0059] In one implementation, the above-mentioned mixed screen request is also used to indicate exiting the mixed screen. Step 104 constructs the mixed screen information according to the screen layout rules based on the mixed screen request, which may include: removing all sub-screens in the current video multi-screen layout; or constructing the mixed screen information according to the sub-screen reduction rules.

[0060] In an embodiment of the present application, when a screen mixing request is received from a terminal indicating exit from screen mixing, the screen mixing information may include removing all existing sub-screens in the current video multi-screen layout, at which point the terminal ends the video live broadcast. It is also possible to choose to construct screen mixing information according to the sub-screen reduction rule, and the screen mixing information may include removing the sub-screen corresponding to the target audio and video stream of the terminal requesting to exit from screen mixing, and moving forward the sub-screen corresponding to the target audio and video stream published by other terminals in the video multi-screen layout; and after removing the sub-screen corresponding to the target audio and video stream of the terminal requesting to exit from screen mixing, the sub-screen corresponding to the audio and video stream to be mixed may be displayed in the vacant position caused by the removal of the sub-screen corresponding to the target audio and video stream, thereby performing corresponding screen mixing processing according to the constructed screen mixing information and updating the video multi-screen layout.

[0061] In one implementation, step 104 of constructing mixed screen information based on the mixed screen request and according to the screen layout rule may include one of the following:

[0062] (1) When the terminal publishes an auxiliary stream of the target audio and video stream and specifies that the sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, the mixed screen information is constructed according to the sub-picture adding rule.

[0063] Among them, when the terminal publishes an auxiliary stream of the target audio and video stream and specifies that the sub-screen corresponding to the auxiliary stream is located at a preset position of the video multi-screen layout, the mixed screen information constructed according to the sub-screen addition rule may include displaying the sub-screen corresponding to the auxiliary stream at the preset position of the video multi-screen layout.

[0064] (2) When the terminal publishes an auxiliary stream of the target audio and video stream and does not specify that the sub-picture corresponding to the auxiliary stream is located at a preset position in the video multi-picture layout, the sub-picture corresponding to the auxiliary stream is not displayed.

[0065] Among them, when the terminal publishes the auxiliary stream of the target audio and video stream and does not specify that the sub-screen corresponding to the auxiliary stream is located at the preset position of the video multi-screen layout, the constructed mixed screen information may include not displaying the sub-screen corresponding to the auxiliary stream.

[0066] (3) When the terminal stops publishing the auxiliary stream of the target audio and video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

[0067] When a terminal stops publishing an auxiliary stream of the target audio and video stream, the mixed screen information constructed according to the sub-screen reduction rule may include removing the sub-screen corresponding to the auxiliary stream, moving the existing sub-screen in the video multi-screen layout forward, or displaying the sub-screen corresponding to the audio and video stream to be mixed in the position vacant after the sub-screen corresponding to the auxiliary stream is removed. This allows the construction of corresponding mixed screen information in different situations based on the mixed screen request initiated by the terminal, and the updating of the video multi-screen layout based on the mixed screen information.

[0068] In one implementation, the above-mentioned screen layout rule may further include: when the terminal publishes multiple target audio and video streams, retaining a sub-screen corresponding to the target audio and video stream in the video multi-screen layout.

[0069] Among them, by limiting the number of sub-screens corresponding to the target audio and video streams released by the display terminal in the video multi-screen layout, the attention to a single target audio and video stream is guaranteed, and a single terminal is prevented from occupying too much of the position of the sub-screen in the video multi-screen layout, so that multiple terminals can participate.

[0070] In one implementation, after receiving the screen mixing request initiated by the terminal in step 102, the method may further include: screening historical audio and video streams released by the terminal.

[0071] In an embodiment of the present application, after receiving a mixing screen request initiated by a terminal, for the historical audio stream published by the terminal, it is determined whether the historical audio stream participates in the current mixing screen and is retained or discarded; and when the historical audio and video stream is not specified to be displayed at a preset position in the video multi-screen layout and the historical audio and video stream is an auxiliary stream, the historical audio and video stream is discarded. By the above-mentioned screening of the historical audio and video streams published by the terminal, unnecessary occupation of the video multi-screen layout by redundant historical audio and video streams is avoided.

[0072] Figure 2 shows a structural schematic diagram of a video multi-screen layout update system provided by an exemplary embodiment of the present application, the system including: a control center and a media processing center; wherein the control center accesses the terminal through a cloud access gateway module 21, and the control center includes an access scheduling module 22, a stream access control module 23, a room control module 24, a stream management module 25, a message control module 26, a message distribution module 27, and a region management service (RMS) module 28; the media processing center includes a stream access negotiation module 29, a media processing service (MPC) module 210, a media forwarding unit (SFU) module 211, a video transcoding synthesis (Video Processor Server, VPS) module 212, a streaming media service (SS) module 213, a media management module 214, and a content delivery network (CDN) module 215.

[0073] Among them, the access scheduling module 22 is used to provide access point allocation services; the stream access control module 23 is used to provide the opening of stream-related capabilities, including stream publishing and subscription, etc., mainly for non-room models; the room control module 24 is used to provide the opening of room-related capabilities, including joining a room, leaving a room, publishing streams in a room, subscribing to streams, etc.; the stream management module 25 is used to publish and subscribe to streams within the real-time audio and video transmission network (RTN), including cascade scheduling of streams;

[0074] The message control module 26 is used to provide the ability to allocate message distribution addresses; the message distribution module 27 is used to provide message subscription and publishing, and cooperate with the message control module 26; the regional management service module 28 is used to provide edge distribution services; the stream access negotiation module 29 is used to provide media negotiation capabilities and link detection capabilities; the media processing service module 210 is used to provide media processing agent services; the media forwarding module 211 is used to provide media forwarding services; the video transcoding and synthesis module 212 is used to provide video transcoding and synthesis services; the streaming media service module 213 is used to provide recording and streaming; the media management and control module 214 is used to provide control services for the media server.

[0075] FIG3 shows a flow chart of another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application. Referring to FIG3 , the method includes the following steps.

[0076] Step 301: The terminal initiates a mixed screen request.

[0077] Step 302: The control center reads the published streams in the room and stores them in a database in order.

[0078] The published flows can be sorted by various rules, such as the flow size, the time when the flow was published, etc.

[0079] Step 303: Filter the published streams.

[0080] Among them, for the audio stream in the published stream, it is determined whether to retain or discard it according to whether the audio stream participates in screen mixing; if the published stream is an auxiliary stream and no preset position is specified, the stream is discarded.

[0081] Step 304: The target streams in the screen mixing request sent by the terminal are stored in order in a list of screens to be mixed.

[0082] The published flows can be sorted by various rules, such as the flow size, the time when the flow was published, etc.

[0083] Step 305: Select a target stream with the maximum number of screens that can be displayed in the current video multi-screen layout from the list of screens to be mixed.

[0084] For example, if the maximum number of screens that can be displayed in the current video multi-screen layout is 9, 9 target streams are selected from the list of screens to be mixed.

[0085] Step 306: Determine the corresponding number of target streams according to the number of idle sub-pictures in the current video multi-picture layout.

[0086] Step 307: construct mixed screen information according to the screen layout rules and store it in a database.

[0087] Step 308: Send the constructed mixed screen information to the media processing center.

[0088] Step 309: The media processing center mixes the target stream published by the terminal according to the mixing information and updates the video multi-screen layout.

[0089] In an embodiment of the present application, after the terminal initiates a mixing screen request, the control center first filters the published streams in the room, then constructs mixing screen information for the target stream and informs the media control center, so that the media control center can mix the target stream published by the terminal according to the mixing screen information, thereby realizing the update of the video multi-screen layout.

[0090] FIG4 shows a flow chart of another method for updating a multi-screen layout of a video provided by an exemplary embodiment of the present application. Referring to FIG4 , the method includes the following steps.

[0091] Step 401: The terminal initiates a mixed screen request, wherein the mixed screen request is used to instruct the terminal to exit the room.

[0092] Step 402: The control center reads the stored mixed screen information.

[0093] Step 403: determine whether the terminal exiting the room is the main initiator of screen mixing based on the stored screen mixing information. If yes, notify the media control center to stop screen mixing; if not, construct screen mixing information according to the sub-screen reduction rule in the screen layout rule.

[0094] Step 404: Send the constructed mixed screen information to the media processing center.

[0095] Step 405: The media processing center removes the target video stream published by the terminal according to the mixed screen information and updates the video multi-screen layout.

[0096] In an embodiment of the present application, when a terminal requests to exit a room, it is necessary to determine whether the terminal is the main initiator of screen mixing. If so, the control center notifies the media control center to stop screen mixing, and the room no longer performs screen mixing. If not, the control center constructs screen mixing information according to the sub-screen reduction rule in the screen layout rule, so that the media control center removes the target video stream published by the terminal according to the screen mixing information, thereby updating the video multi-screen layout. At this time, other terminals can still perform screen mixing in this room, which improves the experience of updating the video multi-screen layout.

[0097] In an exemplary embodiment, an electronic device is also provided, which is used to execute the above-mentioned method for updating the multi-screen layout of the video. Figure 5 is a structural diagram of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call a computer program stored in the memory 503 and run on the processor 501 to execute all or part of the steps of the above-mentioned embodiment of the method for updating the multi-screen layout of the video, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0098] The above electronic device structure does not constitute a limitation of the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. to configure the display panel. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include but are not limited to a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0099] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus random access memory (DRRAM).

[0100] The processor may include one or more processing units; optionally, the processor may integrate an application processor and a modem processor, wherein the application processor primarily handles operations related to the operating system, user interface, and application programs, and the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the processor.

[0101] In an exemplary embodiment, a readable storage medium is also provided, storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps in the above-mentioned method for updating a multi-screen layout of a video. For example, the readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0102] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0103] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for updating a multi-screen layout of a video, wherein: include: Receiving a mixed screen request initiated by a terminal, wherein the mixed screen request is used to instruct the terminal to undergo a preset change; the preset change includes the terminal publishing or stopping publishing a target audio and video stream; Based on the screen mixing request, constructing screen mixing information according to the screen layout rule, wherein the screen mixing information includes information for mixing the sub-screen corresponding to the target audio and video stream with the current video multi-screen layout; The video multi-screen layout is updated based on the mixed screen information.

2. The method according to claim 1, wherein: The screen layout rule includes a rule for adding sub-screens or a rule for reducing sub-screens; wherein, The rule for adding a sub-picture includes one of the following: Displaying the sub-picture corresponding to the target audio and video stream at a preset position of the video multi-picture layout; Displaying the sub-picture corresponding to the target audio and video stream in the first free position of the video multi-picture layout; In the case where there is no free space in the video multi-screen layout, increasing the number of screens in the video multi-screen layout; When the number of pictures in the video multi-picture layout reaches a first threshold, the sub-picture corresponding to the target audio and video stream is not displayed; wherein the first threshold is the number of pictures at the highest level; The sub-picture reduction rule includes at least one of the following: Removing the sub-picture corresponding to the target audio and video stream, and moving the sub-picture that already exists in the video multi-picture layout forward; The sub-screen corresponding to the audio and video stream to be mixed is displayed in the first free position of the video multi-screen layout.

3. The method according to claim 2, wherein: After removing the sub-picture corresponding to the target audio and video stream, the method further includes: When the number of pictures in the video multi-picture layout does not exceed a second threshold, reducing the number of pictures in the video multi-picture layout; The second threshold is a number of pictures less than the highest level.

4. The method according to claim 2, wherein: The constructing the mixed screen information according to the screen layout rule based on the mixed screen request includes one of the following: In the case where the terminal publishes the target audio stream, constructing the mixed screen information according to the rule of adding sub-pictures; In a case where the terminal stops publishing the target audio stream, the mixed screen information is constructed according to the sub-picture reduction rule.

5. The method according to claim 2, wherein: The constructing the mixed screen information according to the screen layout rule based on the mixed screen request includes one of the following: In the case where the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures; In the case where the terminal stops publishing the target video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

6. The method according to claim 5, wherein: When the terminal publishes the target video stream, constructing the mixed screen information according to the rule of adding sub-pictures includes: In the case that the terminal publishes the target video stream and has published the historical audio stream, the sub-picture corresponding to the historical audio stream is replaced with the sub-picture corresponding to the target video stream.

7. The method according to claim 5, wherein: The step of constructing the mixed screen information according to the sub-picture reduction rule when the terminal stops publishing the target video stream includes one of the following: When the terminal stops publishing the target video stream and has published the historical audio stream, replacing the sub-picture corresponding to the target video stream with the sub-picture corresponding to the historical audio stream; When the terminal stops publishing the target video stream and has published the historical video stream, the sub-picture corresponding to the target video stream is replaced with the sub-picture corresponding to the historical video stream.

8. The method according to claim 2, wherein: The mixed screen request is also used to indicate exiting the mixed screen; The constructing mixed screen information based on the mixed screen request and according to the screen layout rule includes: Remove all sub-screens from the current video multi-screen layout; or, The mixed screen information is constructed according to the sub-picture reduction rule.

9. The method according to claim 2, wherein: The constructing the mixed screen information according to the screen layout rule based on the mixed screen request includes one of the following: When the terminal publishes the auxiliary stream of the target audio and video stream and specifies that the sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, constructing the mixed screen information according to the sub-picture adding rule; When the terminal publishes an auxiliary stream of the target audio and video stream and does not specify that the sub-picture corresponding to the auxiliary stream is located at a preset position of the video multi-picture layout, the sub-picture corresponding to the auxiliary stream is not displayed; In a case where the terminal stops publishing the auxiliary stream of the target audio and video stream, the mixed screen information is constructed according to the sub-picture reduction rule.

10. The method according to claim 1, wherein: The screen layout rules also include: In the case where the terminal publishes multiple target audio and video streams, a sub-picture corresponding to the target audio and video stream is reserved in the video multi-picture layout.

11. The method according to claim 1, wherein: After the mixed screen request initiated by the receiving terminal, the method further includes: Filter the historical audio and video streams published by the terminal.

12. An electronic device, wherein: The electronic device includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for updating the video multi-screen layout as described in any one of claims 1 to 11 are implemented.

13. A readable storage medium, wherein: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the method for updating the video multi-screen layout according to any one of claims 1 to 11 are implemented.

Images