Method and apparatus for displaying video of continuous microphone, device and medium

Abstract

The application discloses a method and device for displaying a mixed video stream in a network live broadcast process, and a medium. The method comprises the following steps: acquiring a mixed video stream forwarded by a content distribution network in a network live broadcast process, wherein the mixed video stream contains video images of two or more anchor users in a live broadcast; determining target playing windows to which each anchor user belongs in response to a video layout switching instruction of a viewer user, wherein some target playing windows have a layering relationship; identifying and extracting video frames of the video images of each anchor user in a full frame of the mixed video stream, and extracting the video images of each anchor user according to each video frame; and respectively displaying the video images of each anchor user to the target playing windows corresponding to the anchor users. The application can flexibly adjust the layout of each video image of a mixed video stream in a network live broadcast process.

Description

TECHNICAL FIELD

[0001] The present application relates to the network live broadcast technical field, and particularly relates to a method for displaying a video in a live broadcast, and a corresponding device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In a network live broadcast scenario, a host user pushes a video stream to a live broadcast room to achieve the application purposes of talent show, information sharing, knowledge education and the like, so that the host user can participate in social labor through these activities to obtain benefits and promote overall social benefits.

[0003] For cost control, in a network live broadcast process, the terminal device of a host user participating in a live broadcast can push the live broadcast video streams produced by the host users to one of the terminal devices of the host users or a media server for mixed picture processing, for example, adjacent splicing of video images of two host users, so that the mixed picture video stream obtained through the processing can be forwarded to the terminal device of a viewer user through a content distribution network for display. In this way, the viewer user only needs to obtain a single mixed picture video stream to see the video images of multiple host users in a live broadcast.

[0004] Although the mixed picture video stream can save network traffic and offset the slightly higher latency of the content distribution network relative to a self-built server responsible for forwarding a live broadcast video stream, it brings difficulties to the terminal device in flexibly adjusting the layout of the video images of the host users in the same graphical user interface, and additional technical means are needed to achieve this. Therefore, how to adjust the mixed picture video stream of the host users in a live broadcast in the terminal device and decouple the video images of the host users to achieve flexible layout adjustment has become a problem that the industry hopes to solve. SUMMARY

[0005] The primary purpose of the present application is to solve at least one of the above problems and provide a method for displaying a video in a live broadcast, a corresponding device, an electronic device and a computer readable storage medium.

[0006] To meet the various purposes of the present application, the present application adopts the following technical solutions:

[0007] A method for displaying a video in a live broadcast adapted to one of the purposes of the present application includes the following steps:

[0008] Obtaining a mixed picture video stream forwarded through a content distribution network in a network live broadcast process, the mixed picture video stream containing video images of two or more host users in a live broadcast;

[0009] In response to a video layout switching instruction of a viewer user, determining target playing windows to which the host users defined by the video layout switching instruction belong, and some of the target playing windows having a layering relationship;

[0010] identify video frames occupied by video images of each anchor user in the full frame of the mixed video stream, and extract the video images of each anchor user according to each of the video frames;

[0011] display the video images of each anchor user to the target play window corresponding to each anchor user, respectively.

[0012] In an optional embodiment, identifying the video frames occupied by the video images of each anchor user in the full frame of the mixed video stream comprises:

[0013] extracting at least two adjacent image frames in the mixed video stream;

[0014] calculating frame difference information between the at least two adjacent image frames, and determining the division line information between the video images of each anchor user according to the frame difference information;

[0015] determining the video frames occupied by the video images of each anchor user according to the division line information and the full frame of the mixed video stream.

[0016] In an optional embodiment, calculating the frame difference information between the at least two adjacent image frames, and determining the division line information between the video images of each anchor user according to the frame difference information comprises:

[0017] finding two rows of pixels in the same row in the mixed video stream with the smallest sum of pixel difference values of all pixels in the two rows of pixels based on the frame difference information between each pair of adjacent image frames;

[0018] judging whether the two rows of pixels are successive in position, and using the position of the two rows of pixels to determine the division line information when the two rows of pixels are successive;

[0019] when the division line information is not determined, iteratively determining the division line information using the frame difference information between the next pair of adjacent image frames.

[0020] In an optional embodiment, when the two rows of pixels are successive, using the position of the two rows of pixels to determine the division line information comprises:

[0021] when the two rows of pixels are successive, comparing the position of the two rows of pixels in the current judgment with the position of the two rows of pixels in the last round of judgment, using the position of the two rows of pixels to determine the division line information when the position is unchanged, otherwise, iteratively determining the division line information using the frame difference information between the next pair of adjacent image frames.

[0022] In an optional embodiment, identifying the video frames occupied by the video images of each anchor user in the full frame of the mixed video stream comprises:

[0023] extracting frame ratio information from the supplementary enhancement information of the mixed video stream, the frame ratio information containing the proportion of the video image of each anchor user in the mixed video stream;

[0024] determining the video frame occupied by the video image of each anchor user according to the frame ratio information based on the full frame of the mixed video stream.

[0025] In an optional embodiment, the video image of each anchor user is respectively displayed in the target playback window corresponding to the anchor user, including:

[0026] The display resolution of the video image of each anchor user corresponding to the target playback window is calculated;

[0027] determine whether the display resolution of each video image is lower than the first resolution threshold and higher than the second resolution threshold, and when the determination is true, perform pixel interpolation on the corresponding video image in an interpolation manner to realize quality enhancement;

[0028] determine whether the display resolution of each video image is lower than the second resolution threshold, and when it is lower than the second resolution threshold, perform quality enhancement on the corresponding video image using a preset super-resolution enhancement model;

[0029] The video image after quality enhancement is displayed in the target playback window corresponding thereto.

[0030] In an optional embodiment, after the video image of each anchor user is respectively displayed in the target playback window corresponding to the anchor user, including:

[0031] obtaining an image frame in the mixed video stream;

[0032] segmenting the special effect image of the image frame using a preset image segmentation model;

[0033] determining the special effect type to which the special effect image belongs using a preset special effect classification model;

[0034] execute the local special effect playback instruction corresponding to the special effect type, synchronize the corresponding animation special effect in the current graphical user interface with the image frame, and display the special effect image in the target playback window with the largest image coverage area in the animation special effect.

[0035] A kind of video display device of being connected to the microphone provided for one of the purposes of the present application, comprising:

[0036] The video streaming module is configured to acquire a mixed video stream forwarded by a content distribution network during a network live broadcast process, wherein the mixed video stream comprises video images of two or more anchor users in a live broadcast;

[0037] The switching control module is configured to determine target playing windows to which each anchor user belongs in response to a video layout switching instruction of a viewer user, wherein at least two of the target playing windows have a layering relationship.

[0038] The image extraction module is configured to identify and extract video frames occupied by the video images of each anchor user in a full frame of the mixed video stream, and extract the video images of each anchor user according to each video frame.

[0039] The image display module is configured to display the video images of each anchor user in the target playing window corresponding to each anchor user.

[0040] An electronic device is provided to adapt to one of the purposes of the present application, comprising a central processing unit and a memory, wherein the central processing unit is configured to invoke and run a computer program stored in the memory to perform the steps of the live video display method described in the present application.

[0041] A computer readable storage medium is provided to adapt to another purpose of the present application, which stores a computer program implemented according to the live video display method in the form of computer readable instructions, and the computer program is invoked and run by a computer to perform the steps included in the method.

[0042] A computer program product is provided to adapt to another purpose of the present application, comprising a computer program / instruction, which is executed by a processor to implement the steps of the method described in any one of the embodiments of the present application.

[0043] Compared with the prior art, the present application identifies the video frames occupied by the adjacent video images of each anchor user in the mixed video stream distributed via the content distribution network, accurately extracts the video images of each anchor user from the mixed video stream according to the video frames, and displays the video images of each anchor user in the target playing window corresponding to each anchor user in response to the video layout switching instruction triggered by the user, wherein some of the target playing windows have a layering relationship. Thus, the multiple video images originally adjacent to each other are changed into a layout having a layering relationship, which can present a full screen large window display of one video image and a small window display of another video image. The viewer user does not need to independently pull the live video streams of each anchor user, and the display layout of the video images of each anchor user can be flexibly adjusted in the terminal device receiving the mixed video stream while keeping the minimum consumption of network traffic. Attached Figure Description

[0044] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 This is an exemplary network architecture used for the live streaming service in the live streaming scenario of this application;

[0046] Figure 2 This is a flowchart illustrating one embodiment of the video conferencing display method of this application;

[0047] Figure 3 This is an example diagram illustrating the effect of the original video layout of the mixed video stream in an embodiment of this application;

[0048] Figure 4 This is an example diagram illustrating the effect of switching the video layout of the mixed video stream in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram illustrating the process of determining the video frame of each broadcaster user's video image using the frame difference information between two adjacent image frames in an embodiment of this application.

[0050] Figure 6 This is a flowchart illustrating the process of determining the video frame boundary information based on frame difference information in an embodiment of this application.

[0051] Figure 7 This is a schematic diagram illustrating the process of enhancing the image quality of video images extracted from a mixed video stream in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram illustrating the process of identifying special effects images in a mixed video stream in an embodiment of this application to perform local playback of the corresponding animation effects.

[0053] Figure 9 This is a schematic block diagram of the video display device for live streaming according to this application;

[0054] Figure 10 This is a schematic diagram of the structure of an electronic device used in this application. Detailed Implementation

[0055] Please see Figure 1An exemplary application scenario of the network architecture of the present application includes terminal devices 801, 802, a media server 81, an application server 82, an internal content distribution network 83, and an external content distribution network 84. The terminal devices 801, 802 can be used to run a live room terminal program for a host user or a viewer user to use a live room function, such as uploading a live video stream by the host user through the terminal device 801 to the media server 81, or pushing a target user's live video stream to the host user or the viewer user through the media server 81 to play in the terminal device 802 via the internal content distribution network 83 or the external content distribution network 84, etc. The media server 81 can be responsible for mixing and drawing the live video streams of different host users in a live streaming activity, and then pushing the mixed and drawn video stream to each terminal device 801, 802 of each viewer user in the live room of each host user via the internal content distribution network 83 or the external content distribution network 84. The application server 82 can be used to deploy a network live streaming service to maintain the interaction between the host user and the viewer user based on the live room.

[0056] In one embodiment, the internal content distribution network 83 can be dedicated to the host user, and the external content distribution network 84 can be dedicated to the viewer user, for example, forwarding the live video stream of a second host user to a first host user in a live streaming activity, thereby facilitating the first host user to mix and draw its own live video stream and the live video stream of the second host user to form a mixed and drawn video stream after video image splicing, and pushing the mixed and drawn video stream to the media server 81 for distribution to the viewer user via the internal content distribution network 83 or the external content distribution network 84. In this embodiment, since the internal content distribution network 83 generally has lower latency than the external content distribution network 84, it can serve the generation and forwarding of the video stream on the host user side to ensure low latency, while the external content distribution network 84 has the advantage of low-cost concurrency for a large number of viewer users, so the two complement each other.

[0057] In another embodiment, either the internal content distribution network 83 or the external content distribution network 84 can be used alone to serve the forwarding of the live video stream or the mixed and drawn video stream of the host user to the viewer user.

[0058] It should be noted that whether the mixed and drawn video stream is processed by the terminal device 801 of the host user or by the media server 81 does not affect the embodiment of the spirit of the present application.

[0059] The computer program product programmed to implement the method of displaying a video stream of a mixed video according to the present application can run in the terminal device of the audience user, and through the running of the computer program product, each step of the method is executed, and the technical solution of the present application is implemented, so that the video images of each anchor user in the mixed video stream obtained from the content distribution network are displayed after being rearranged, regardless of whether the content distribution network is an internal content distribution network or an external content distribution network.

[0060] Based on the reference to the above exemplary scenarios and related principle description, please refer to Figure 2 The method of displaying a video stream of a mixed video according to the present application in one embodiment includes the following steps:

[0061] Step S1100, obtaining a mixed video stream forwarded by a content distribution network in a network live broadcast process, the mixed video stream containing video images of two or more anchor users in a live broadcast mode;

[0062] In the network live broadcast process, an anchor user can invite other anchor users to participate in a live broadcast mode. Taking two anchor users participating in the same live broadcast mode as an example, a first anchor user as an initiator invites a second anchor user as a receiver to participate, and the second anchor user enters the live broadcast mode in response to the invitation of the first anchor user. In the live broadcast mode, the live broadcast rooms of the two anchor users are switched to a state of simultaneously displaying live video streams of the two anchor users, and the audience users can simultaneously see the video images of the two anchor users.

[0063] In order to achieve the purpose that the audience users in the live broadcast rooms of the two anchor users can see that the two anchor users appear simultaneously on a graphical user interface, the live video streams of the two anchor users are transmitted to a media server of the network live broadcast. The live video streams of the two anchor users can be mixed and combined into a same live video stream, i.e., a mixed video stream, in the terminal device of any anchor user or in the media server. In the mixed video stream, the live video streams of the two anchor users are cropped and spliced, generally adjacent to each other, so that the mixed video stream can present a video image layout effect as shown in Figure 3 when played, i.e., the video images of each anchor user are displayed side by side in the same mixed video stream.

[0064] When a large number of anchor users, for example, three or more anchor users, appear in the live broadcast mode, the mixed video processing can also be performed in the above manner, and finally a single mixed video stream corresponding to a live broadcast mode is synthesized and pushed to the terminal devices of the audience users in the live broadcast rooms of the anchor users participating in the live broadcast mode, so that all the audience users in each live broadcast room can see the same video image content.

[0065] After the media server obtains or generates the mixed video stream, the mixed video stream can be forwarded to the audience user through a content distribution network. In an embodiment, the background architecture for serving the network live broadcast includes an internal content distribution network and an external content distribution network. The internal content distribution network is used to distribute the live video stream belonging to the anchor user between the media server and the anchor user, and the external content distribution network is used to forward the mixed video stream to the audience user. In order to distinguish the respective characteristics of the internal and external content distribution networks, the live video stream generation link on the anchor user side is ensured to have higher speed, and the mixed video stream pushing link on the audience user side is ensured to have higher cost benefit. Of course, in other embodiments, the anchor user side and the audience user side can share the same content distribution network to forward the live video stream and the mixed video stream, which can be flexibly deployed as needed.

[0066] When the terminal device on the anchor user side responsible for generating the mixed video stream or the media server obtains the live video streams of different anchor users participating in the same co-hosting activity, the video images in the live video streams of each anchor user can be cropped according to a certain full-frame size, and then the video images are spliced into the mixed video stream in a certain order.

[0067] Of course, in other embodiments, adaptive image enhancement processing can also be performed on the video images of each anchor user before splicing to ensure that the mixed video stream has high video quality.

[0068] When the audience user enters the live broadcast room where the co-hosting activity is being held, the audience user can continuously pull the mixed video stream from the content distribution network, and then display the mixed video stream to the graphical user interface of the terminal device.

[0069] In step S1200, in response to the video layout switching instruction of the audience user, the target play window to which each anchor user belongs defined by the video layout switching instruction is determined, and some target play windows have a stacking relationship.

[0070] When the user needs to switch the layout relationship between the video images of different anchor users in the mixed video stream, the corresponding video layout switching instruction can be issued. For example, the video layout switching instruction can be triggered by a specific touch gesture or by a control button provided in the graphical user interface. When the video layout switching instruction is triggered, the background process of the live broadcast program responds to the switching instruction, determines the layout information representing the video layout corresponding to the switching instruction, and obtains the target play window corresponding to each anchor user participating in the co-hosting activity provided in the layout information.

[0071] The playing relationship of the target playing window of each anchor user can be defined as containing a layering relationship in the present application, for example, for a terminal device such as a smart phone, the target playing window of a first anchor user participating in a live-mixing activity can be set as a full-screen size, and the target playing window of one or more other anchor users can be set as a small window layered above the target playing window of the first anchor user. Figure 4 An example of a layering relationship of the target playing window of two anchor users is shown for reference. In the data layer, the background process of the live broadcast program obtains the position information of the playing window of each anchor user in the graphical user interface through the layout information, which can specifically be expressed as corresponding coordinate information.

[0072] In an embodiment, the correspondence between each anchor user and the target playing window thereof can be processed in the following manner: for an anchor user that the current audience user has followed or that is the anchor user of all people in the live broadcast room in which the current audience user is located, the anchor user is defaulted as a first anchor user, and the video image thereof can be displayed in a full-screen target playing window; one or more other anchor users can be randomly assigned one other target playing window. Generally, when generating a mixed video stream, the video image of the first anchor user can be placed in a default position, for example, the left side, to facilitate the identification of the user identity to which the video image belongs on the audience user side.

[0073] In addition, in other embodiments, the process of the live broadcast program can respond to a video image interchanging instruction between each two target playing windows that is acted on by the audience user, to realize the interchanging of the mapping relationship between the anchor users to which the two target playing windows belong. And so on, a person skilled in the art can flexibly set as needed to improve the fine operation of the audience user to change the video image display layout between multiple anchor users.

[0074] Step S1300, identifying the video frame occupied by the video image of each anchor user in the full frame of the mixed video stream, and extracting the video image of each anchor user according to each video frame;

[0075] In order to enable each target playing window to display the video image of the corresponding anchor user, it is necessary to identify the video frame occupied by the video image of each anchor user in the mixed video stream, and on the basis of identifying the video frame occupied by each anchor user in the full frame of the mixed video stream, the video image of each anchor user corresponding to each video frame is extracted from the mixed video stream for separate processing of these video images.

[0076] One of the ways to identify the video frame occupied by the video image of each anchor user in the full frame of the mixed video stream can be provided by the terminal device or the media server responsible for the mixed processing of the video image of each anchor user with its corresponding frame ratio information hidden in the mixed video stream for transmission, so that the audience user side can calculate the video frame occupied by the video image of each anchor user according to the frame ratio information.

[0077] In another way to identify the video frame occupied by the video image of each anchor user in the full frame of the mixed video stream, without transmitting any splicing information about the video image through the mixed video stream, the live program process in the terminal device receiving the mixed video stream finds out the division line information between the video images of each anchor user based on two consecutive image frames in the mixed video stream, and then divides the video frame corresponding to the video image corresponding to each anchor user according to the division line information.

[0078] It is not difficult to understand that by identifying the video frame of each anchor user through the mixed video stream, the video images of different anchor users can be distinguished and divided into different independent video images, and the layout of these video images can be adjusted.

[0079] Step S1400, draw and display the video images of the respective anchor users to the target play window corresponding to each anchor user.

[0080] After extracting the video images of each anchor user from the mixed video stream, according to the corresponding relationship between the video images of each anchor user and the target play window of each anchor user, draw and display each video image to the target play window corresponding thereto respectively, realize the play effect, so that the audience user can watch the synchronous play effect of the video images of multiple anchor users displayed in the new video layout in the graphical user interface.

[0081] In one embodiment, when drawing and displaying each video image to the target play window corresponding thereto, refer to the video frame ratio information of each anchor user hidden in the mixed video stream, and calculate the video frame occupied by the video image of each anchor user according to the video frame ratio information. Figure 4, the following process can be performed: first, the size of the entire target playing window is specified by an opengl instruction, which is the display area of the first anchor user and is defined as window 1; then, the position of the area of the target playing window to be scaled of the second anchor user is obtained, which is defined as window 2; further, the left first anchor user texture coordinate TexLeft and the right second anchor user texture coordinate TexRight are obtained according to the process of step S1300; then, the texture of the left first anchor user is sampled according to the texture coordinate TexLeft to obtain the video image of the first anchor user; then, the scaling algorithm is performed on the right second anchor user texture coordinate TexRight to scale to Figure 4 the texture area corresponding to the second anchor user, i.e., window 2; then, the right second anchor user is sampled according to the scaled texture coordinate to obtain the scaled video image of the second anchor user; further, the pixels in different target playing windows are output, the image pixels corresponding to the second anchor user are output in window 2, and the image pixels corresponding to the first anchor user are output in the area of window 1 except the position containing window 2; finally, the result output in the previous step is displayed on the screen.

[0082] In some embodiments, for the video image enlarged from a smaller display scale to a larger display scale, i.e., the target playing window of the anchor user is larger than the display scale before the video layout switching instruction is implemented, whether image interpolation or super-resolution enhancement processing is performed on the enlarged video image to improve the quality of the enlarged video image can be determined according to whether the resolution of the enlarged video image meets the preset condition.

[0083] According to the above embodiments, for the mixed video stream distributed via the content distribution network, the video frames occupied by the adjacent video images of each anchor user are identified, the video images of each anchor user are accurately extracted from the mixed video stream according to the video frames, and when the user triggers the video layout switching instruction, the video images of each anchor user are displayed in the target playing window corresponding to each anchor user. Some of these target playing windows have a stacking relationship, thereby changing the originally adjacent multiple video images into a layout with a stacking relationship, which can present, for example, one video image as a full-screen large window display and another video image as a small window display. The audience user side does not need to independently pull the live video stream of each anchor user, and the display layout of the video images of each anchor user can be flexibly adjusted in the terminal device receiving the mixed video stream while keeping the network traffic consumption at a minimum. In addition, the network live background architecture can be separated from the internal and external content distribution networks, and the advantages of balancing the time delay and cost of the two content distribution networks can be achieved.

[0084] On the basis of any embodiment of the present application, refer to Figure 5 , identify the video frame occupied by the video image of each anchor user in the full frame of the mixed video stream, comprising:

[0085] Step S2100, extract at least two adjacent image frames in the mixed video stream;

[0086] In order to identify the video frame where the video image of each anchor user is located from the mixed video stream, each two adjacent image frames in the mixed video stream can be used as material, and subsequent processes are applied for analysis. Generally, during the broadcasting process of the anchor user, the position of the camera and the ambient light are relatively fixed, so the boundary information in the video stream image after imaging is relatively fixed. Even if the video image is cropped during mixing, the boundary information is relatively fixed in each two adjacent image frames. Therefore, the boundary of the video image of the anchor user can be identified according to this principle.

[0087] Step S2200, calculate the frame difference information between the at least two adjacent image frames, and determine the dividing line information between the video images of each anchor user according to the frame difference information;

[0088] For the two image frames contained in each two adjacent image frames, the two image frames are subtracted according to one-to-one correspondence of pixel points to obtain the frame difference information, and the difference value of each pixel point in the full frame range of the mixed video stream can be obtained. It is not difficult to understand that, taking the video layout of the splicing of the two anchor users shown in Figure 3 As an example, in the left and right video images, the difference value in the frame difference information of the pixel points in one row at the rightmost boundary of the left video image and the pixel points in one row at the leftmost boundary of the right video image is generally relatively stable, and the difference value is relatively the lowest. Therefore, according to this principle, the pixel in one row at the right boundary of the left video image and the pixel in one row at the left boundary of the right video image are found out, which actually finds out the dividing line information of the video images of different anchor users.

[0089] Step S2300, determine the video frame occupied by the video image of each anchor user according to the dividing line information and the full frame of the mixed video stream.

[0090] After the dividing line information is determined, each image frame of the mixed video stream can be delimited by the dividing line information, and the video frame occupied by the video image of each anchor user is determined. The video frame can be represented as coordinate position information, for example, the coordinate position information of the video frame occupied by each video image can be represented by the coordinates of the upper left corner and the lower right corner.

[0091] For the case that more than three anchor users' video images are adjacent spliced in a more complex position manner in the mixed video stream, the interface line information can also be found in the above manner, mainly embodied as finding a plurality of sets of two adjacent rows of pixels, and then determining the video frame of each anchor user's video image by using the layout relationship of the plurality of sets of two rows of pixels in the plane provided by the full frame of the mixed video stream.

[0092] According to the above embodiments, it can be understood that the frame difference information is used to find the interface line information of different anchor users' video images in the mixed video stream, and then the video frame occupied by each anchor user's video image is determined according to the interface line information, which has low computational complexity, accurate and rapid demarcation, and does not need to rely on external information, and is not affected even if individual frame loss occurs in the transmission process of the mixed video stream.

[0093] Based on any embodiment of the present application, please refer to Figure 6 , the frame difference information between the at least two adjacent image frames is calculated, and the interface line information between each anchor user's video image is determined according to the frame difference information, including:

[0094] Step S2210, finding two rows of pixels with the smallest sum of pixel difference values of all pixels in the same row in the mixed video stream based on the frame difference information between each pair of adjacent image frames;

[0095] Taking the video layout of the mixed video stream shown in Figure 4 as an example, when the audience user side starts to receive the mixed video stream, the first two image frames are extracted first, the difference values of each corresponding pixel point between the two image frames are calculated, and the frame difference information between the two image frames is constructed.

[0096] Then, in the frame difference information, the sum of the difference values of each row and each column of pixel points is calculated in units of rows, and two rows of pixels with the smallest sum of difference values are determined. Generally, the two rows of pixels should be in the same direction, for example, in the same row direction or in the same column direction.

[0097] Step S2220, judging whether the two rows of pixels are consecutive in position, and when the two rows of pixels are consecutive, using the positions of the two rows of pixels to determine the interface line information;

[0098] Further, according to the position coordinates of the two rows of pixels, it is judged whether the two rows of pixels are consecutive, and if they are consecutive, the interface line information can be determined by using the positions of the two rows of pixels. The interface line information can be represented as the direction coordinate values of the two rows of pixels, as long as it is convenient for subsequent calculation.

[0099] In some embodiments, a decision threshold can be further set, and the sum of the differences of the two rows of pixels determined initially is compared with the decision threshold, and when one or both of the sums of the differences of the two rows of pixels is less than the decision threshold, it is considered that the boundary line information is determined, which can ensure more accurate boundary identification effect.

[0100] Step S2230, when the boundary line information is not determined, the frame difference information between the next pair of adjacent image frames is used to determine the boundary line information iteratively.

[0101] If the two rows of pixels are not the two rows of pixels in succession, it indicates that the two rows of pixels are not enough to determine the boundary line information, and then the next pair of adjacent image frames can be taken to continue to calculate the frame difference information to determine the boundary line information as soon as possible, and the video frame of each anchor user can be determined according to the boundary line information.

[0102] Of course, after the boundary line information is determined and the video frame is determined, the above process can be continued iteratively, and then the boundary line information found again is compared with the boundary line information determined, and when the boundary line information changes, the video frame of each anchor user is adjusted again according to the new boundary line information, and the video image of each anchor user is repositioned, so that the video image partition of each anchor user can be tracked and adjusted in real time.

[0103] Therefore, in one embodiment, after the judgment in step S2220 is performed, when the two rows of pixels are in succession, the positions of the two rows of pixels determined in the judgment are compared with the positions of the two rows of pixels determined in the last round of judgment, when the positions are the same, the positions of the two rows of pixels are used to determine the boundary line information, otherwise, step S2230 is executed to continue to determine the boundary line information iteratively using the frame difference information between the next pair of adjacent image frames.

[0104] According to the above embodiments, it can be understood that the boundary line information of the video image of each anchor user in the mixed video stream is found based on the frame difference information, and the video frame occupied by the video image of each anchor user is determined according to the boundary line information, which has low calculation amount, accurate and rapid boundary, and does not need to rely on external information. Even if individual frames are lost during the transmission of the mixed video stream, it is not affected, and the video frame occupied by each anchor user can be accurately determined to adapt to the dynamic change of the video frame of each anchor user in the mixed video stream, and the video image of each anchor user is accurately cut.

[0105] On the basis of any embodiment of the present application, the video frame occupied by the video image of each anchor user in the full frame of the mixed video stream is identified and extracted, including:

[0106] Step S3100, extracting frame ratio information from the supplementary enhancement information of the mixed video stream, the frame ratio information containing the proportion of the video image of each anchor user in the mixed video stream;

[0107] The device responsible for implementing the mixed picture, such as the terminal device of the first anchor user or the media server, when implementing the mixed picture of the video images of each anchor user participating in the same co-broadcasting activity, splices the video images of each anchor user according to a certain frame ratio, so that the frame ratio information can be transmitted to the audience user side together with the mixed video stream, so that the audience user side can calculate the video frame occupied by the video image of each anchor user according to the frame ratio information.

[0108] In an embodiment, to avoid increasing the transmission channel, the frame ratio information can be added to the additional enhancement information (SEI) of the mixed video stream when transmitting the mixed video stream by using the convenience provided by related video protocols such as H.264, H.265, and the like.

[0109] Step S3200, based on the full frame of the mixed video stream, calculating and determining the video frame occupied by the video image of each anchor user according to the frame ratio information.

[0110] After the terminal device of the audience user side receives the mixed video stream, the full frame information, i.e., the size information, of the frame ratio information is read from the additional enhancement information thereof, and the video frame of the video image of each anchor user can be calculated and determined according to the proportion of the video image of each anchor user in the full frame specified by the frame ratio information.

[0111] According to the above embodiments, the frame ratio information can be embedded in the mixed video stream by the device responsible for the mixed picture and pushed to the audience user side, and the terminal device of the audience user side can directly calculate and determine the video frame occupied by the video image of each anchor user by using the frame ratio information, which is relatively fast.

[0112] On the basis of any embodiment of the present application, please refer to Figure 7 displaying the video images of the anchor users to the target play windows corresponding to the anchor users, respectively, including:

[0113] Step S4100, calculating the display resolution of the video image of each anchor user corresponding to the target play window corresponding thereto;

[0114] When the video images of each anchor user are drawn and displayed to the corresponding target playing window, scaling usually occurs, and the quality of the display after scaling can be identified by the display resolution after scaling. Thus, for each video image generated by the matting from the mixed video stream, the display resolution after drawing to the target playing window can be calculated according to the size information of the target playing window. It is not difficult to understand that for the scaling operation of enlarging the video image, the display resolution of the video image will usually be reduced; conversely, for the scaling operation of reducing the video image, the video image may be excessively reduced to cause jaggy after display, and the impact of the enlargement operation is more obvious, and thus should be paid more attention to.

[0115] Step S4200, judging whether the display resolution of each video image is lower than the first resolution threshold and higher than the second resolution threshold, and when the judgment is true, performing pixel interpolation on the corresponding video image in an interpolation manner to realize quality enhancement;

[0116] Further, the display resolution of each video image is judged to determine whether it is lower than the first resolution threshold and higher than the second resolution threshold, wherein the second resolution threshold is lower than the first resolution threshold. When the judgment is true, it indicates that the corresponding scaling operation will cause image distortion, but the distortion degree is relatively limited. In this case, the video image can be enhanced by image enhancement processing in a pixel interpolation manner to enhance the quality of the video image. The pixel interpolation manner can be implemented by linear or nonlinear interpolation, and when linear interpolation is adopted, bicubic interpolation algorithm or other arbitrary algorithms can be selected and used by those skilled in the art.

[0117] Step S4300, judging whether the display resolution of each video image is lower than the second resolution threshold, and when it is lower than the second resolution threshold, a preset super-resolution enhancement model is used to enhance the quality of the corresponding video image;

[0118] The display resolution is continuously judged to determine whether it is lower than the second resolution threshold. When it is lower than the second resolution threshold, it indicates that the distortion degree of the scaled video image is higher, and ordinary interpolation manner may be difficult to obtain better quality enhancement effect. Thus, a super-resolution enhancement model trained to convergence in advance can be used to enhance the quality of the video image to greatly improve the video quality of the scaled video image.

[0119] Step S4400, drawing and displaying the video image after quality enhancement to the corresponding target playing window.

[0120] The video images of each anchor user are enhanced in quality according to the display resolution thereof, and then are displayed in the corresponding target playing window according to the method described above.

[0121] As can be seen from the above embodiments, by distinguishing the video images of different anchor users, calculating the display resolution thereof after scaling respectively, and performing quality enhancement processing on the corresponding video images in different ways based on display resolutions of different levels using the corresponding relationship between display resolution and quality, it can be ensured that after adjusting the video layout, the video images of each anchor user participating in the live activity can obtain good quality effect, and the user viewing experience is good.

[0122] Based on any embodiment of the present application, please refer to Figure 8 After the video images of each anchor user are displayed in the corresponding target playing window respectively, the method further comprises:

[0123] Step S1500, acquiring an image frame in the mixed video stream;

[0124] In a live room, various animation effects are often triggered, and the animation effects are often synthesized into a live video stream. The animation effects may be synthesized in a live video stream of a single anchor user or in a full-frame picture of a mixed video stream. The former may be cropped in the mixed picture stage, and the latter may be cropped when the video images of different anchor users are segmented on the side of a viewer user. Regardless of how, when the full frame of the mixed video stream is segmented, the originally normally displayed animation effects are likely to be displayed incompletely in the adjusted video layout. To this end, the technical means of the present application can be used to overcome this problem, and the basis is to acquire image frames in the mixed video stream frame by frame and perform subsequent processing based on the image frames.

[0125] Step S1600, dividing a special effect image of the image frame using a preset image segmentation model;

[0126] In order to identify the special effect image in the mixed video stream, a corresponding training sample is used to train an image segmentation model in advance. The image segmentation model can be constructed and trained based on a U-net series of deep learning models, so that it learns to determine the image mask of various special effect images in a given image frame to obtain the position information of the special effect image relative to the image frame.

[0127] Accordingly, the image frame is input into the image segmentation model, and the image mask corresponding to the special effect image in the image frame is segmented by the image segmentation model, and in fact, the position information of the special effect image in the image frame is obtained. Since the position information has correspondence in the image frame and in the video image of the anchor user extracted based on the image frame, the display position of the special effect image in the new video layout after the video layout adjustment can be determined by using the correspondence and the image mask. The display position is usually determined by the position of the special effect image in the target playback window with the largest display area after the video layout adjustment.

[0128] In addition, the image segmentation of the image frame based on the image mask can also obtain the special effect image.

[0129] Step S1700, determine the special effect type to which the special effect image belongs by using a preset special effect classification model;

[0130] In order to identify which animation special effect the special effect image belongs to, a special effect classification model can also be trained in advance, and a mapping relationship data between each special effect image of the animation special effect and the special effect type is used to construct a training sample and a supervised label pair for training the special effect classification model, so that the special effect classification model learns the ability to determine the corresponding special effect type according to the input special effect image. The special effect classification model can be a neural network-based deep learning model.

[0131] Accordingly, for the special effect image segmented in the previous step, inputting the special effect image into the special effect classification model can determine the special effect type to which the special effect image belongs.

[0132] Step S1800, execute the local special effect playback instruction corresponding to the special effect type, and play the corresponding animation special effect in the current graphical user interface synchronously with the image frame, and cover the special effect image in the image in the target playback window with the largest image display area.

[0133] The live program has local special effect playing instructions corresponding to each special effect type predefined locally. According to the special effect type determined in the previous step, the local special effect playing instruction corresponding to the special effect type is called and executed, and the display position of the special effect image in the current video layout is given by the local special effect playing instruction. As a result, the local special effect playing instruction is executed, the corresponding animation special effect is played, the playing position is aligned with the display position, and the image frame is played synchronously. Thus, the effect obtained is that the special effect image of the animation special effect played locally can cover the same special effect image appearing in the target playing window with the largest display area. Since the animation special effect is replayed in the local graphical user interface and can cover the original special effect image in the largest display area, a better special effect playing effect is obtained, and the special effect playing process is more smooth and natural.

[0134] According to the above embodiments, after adjusting the video layout of the video images of each anchor user in the mixed video stream, the special effect image in the mixed video stream is accurately identified, the corresponding special effect type of the special effect image is determined, and the corresponding local special effect playing instruction is called and executed according to the special effect type. The pre-synthesized special effect image in the mixed video stream can be reproduced with high quality, and the user experience is good.

[0135] Please refer to Figure 9 , one of the purposes of the present application is to provide a live video display device, which comprises a video streaming module 1100, a switching control module 1200, an image extraction module 1300, and an image display module 1400. The video streaming module 1100 is configured to obtain a mixed video stream forwarded by a content distribution network during a network live broadcast process, and the mixed video stream contains video images of two or more anchor users in a live video. The switching control module 1200 is configured to determine target playing windows to which each anchor user belongs in response to a video layout switching instruction of a viewer user, wherein at least two target playing windows have a layering relationship. The image extraction module 1300 is configured to identify and extract video frames occupied by video images of each anchor user in the full frame of the mixed video stream, and extract the video images of each anchor user according to each video frame. The image display module 1400 is configured to draw and display the video images of each anchor user to the corresponding target playing window of each anchor user.

[0136] On the basis of any embodiment of the present application, the image extraction module 1300 comprises: a frame pair extraction unit configured to extract at least two adjacent image frames in the mixed video stream; a boundary analysis unit configured to calculate frame difference information between the at least two adjacent image frames, and determine boundary line information between video images of each anchor user according to the frame difference information; and a frame division unit configured to determine video frames occupied by the video images of each anchor user according to the boundary line information and a full frame of the mixed video stream.

[0137] On the basis of any embodiment of the present application, the boundary analysis unit comprises: a boundary finding subunit configured to find two rows of pixels with a minimum sum of pixel difference values of all pixels in the same row in the mixed video stream based on frame difference information between each pair of adjacent image frames; an interface discrimination subunit configured to determine whether the two rows of pixels are successive in position, and determine the boundary line information using positions of the two rows of pixels when the two rows of pixels are successive; and an iterative processing subunit configured to determine the boundary line information iteratively using frame difference information between a next pair of adjacent image frames when the boundary line information is not determined.

[0138] On the basis of any embodiment of the present application, the interface discrimination subunit comprises: when the two rows of pixels are successive, comparing positions of the two rows of pixels determined in this time with positions of the two rows of pixels determined in a last time, determining the boundary line information using positions of the two rows of pixels when the positions are unchanged, or otherwise, continuing to determine the boundary line information iteratively using frame difference information between a next pair of adjacent image frames.

[0139] On the basis of any embodiment of the present application, the image extraction module 1300 comprises: a proportion acquisition unit configured to extract frame proportion information from supplementary enhancement information of the mixed video stream, the frame proportion information comprising proportions of the video images of each anchor user in the mixed video stream; and a frame determination unit configured to determine video frames occupied by the video images of each anchor user according to the frame proportion information based on a full frame of the mixed video stream.

[0140] On the basis of any embodiment of the present application, the image display module 1400 comprises: a picture quality analysis unit configured to calculate the display resolution of each anchor user's video image corresponding to the corresponding target play window; an interpolation enhancement unit configured to determine whether the display resolution of each video image is lower than the first resolution threshold and higher than the second resolution threshold, and when the determination is true, performing pixel interpolation on the corresponding video image in an interpolation manner to realize picture quality enhancement; a depth enhancement unit configured to determine whether the display resolution of each video image is lower than the second resolution threshold, and when it is lower than the second resolution threshold, performing picture quality enhancement on the corresponding video image by using a preset super-resolution enhancement model; and a display processing unit configured to display the picture quality enhanced video image to the corresponding target play window.

[0141] On the basis of any embodiment of the present application, the application further provides a live video display device, which comprises: an image frame calling module configured to obtain image frames in the mixed picture video stream; an special effect image segmentation module configured to segment out special effect images of the image frames by using a preset image segmentation model; an special effect type determination module configured to determine the special effect type to which the special effect images belong by using a preset special effect classification model; and an special effect local play module configured to execute a local special effect play instruction corresponding to the special effect type, play a corresponding animation special effect in the current graphical user interface synchronously with the image frames, and display the special effect images in the target play window in which the image in the animation special effect has the largest image overlay display area.

[0142] To solve the above technical problems, the present application further provides an electronic device. As shown in the Figure 10 internal structure diagram of the electronic device. The electronic device comprises a processor, a computer readable storage medium, a memory and a network interface connected by a system bus. The computer readable storage medium of the electronic device stores an operating system, a database and computer readable instructions. The database can store control information sequences. When the computer readable instructions are executed by the processor, the processor can implement a live video display method. The processor of the electronic device is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory of the electronic device can store computer readable instructions. When the computer readable instructions are executed by the processor, the processor can execute the live video display method of the present application. The network interface of the electronic device is used to communicate with the terminal. Those skilled in the art can understand that Figure 10 the structure shown in the figure is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the electronic device to which the present application scheme is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0143] The processor in the embodiment is configured to execute the specific functions of each module and unit in the embodiment, and the memory stores the program codes and various data required for executing the modules or units. The network interface is configured to transmit data between the user terminal and the server. The memory in the embodiment stores the program codes and data required for executing all the modules / units in the live video display device of the present application, and the server can call the program codes and data of the server to execute the functions of all the units. Figure 9

[0144] The present application also provides a storage medium storing computer readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the live video display method of any embodiment of the present application.

[0145] The present application also provides a computer program product, including computer programs / instructions, which, when executed by one or more processors, implement the steps of the method described in any embodiment of the present application.

[0146] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. The storage medium can be a computer readable storage medium, such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0147] In summary, the present application can flexibly adjust the layout of the video images of each anchor user in the mixed video stream received by the user side in the network live broadcast process, which not only saves network traffic, but also helps to reflect the cost advantage of the network live broadcast background deployment, and can improve the user experience of watching the live video stream generated by live voice.​

Claims

1. A method for displaying live video chat, characterized in that, Includes the following steps: Acquire a mixed video stream that is forwarded by a content distribution network during a live online broadcast, wherein the mixed video stream contains video images of two or more live broadcasting users who are connected in the same room. In response to the video layout switching command from the audience user, the target playback window of each anchor user defined by the video layout switching command is determined, and some target playback windows have a stacking relationship. Identify and extract the video frame area occupied by each anchor user's video image in the full frame of the mixed video stream, and extract the video image of each anchor user based on each video frame area; The video images of each broadcaster user are drawn and displayed in the corresponding target playback window of each broadcaster user; Obtain image frames from the mixed video stream; The special effects image of the image frame is segmented using a preset image segmentation model; The special effect type of the special effect image is determined by using a preset special effect classification model; Execute the local effect playback instruction corresponding to the effect type, play the corresponding animation effect synchronously with the image frame in the current graphical user interface, and cover the effect image appearing in the target playback window with the largest display area with the image in the animation effect.

2. The video display method for live streaming according to claim 1, characterized in that, Identify and extract the video frame size occupied by each broadcaster's video image in the full frame of the mixed video stream, including: Extract at least two adjacent image frames from the mixed video stream; Calculate the frame difference information between the at least two adjacent image frames, and determine the boundary information between the video images of each broadcaster user based on the frame difference information; Based on the boundary line information and the full frame of the mixed video stream, the video frame occupied by the video image of each broadcaster user is determined.

3. The video display method for live streaming according to claim 2, characterized in that, Calculate the frame difference information between at least two adjacent image frames, and determine the boundary information between the video images of each broadcaster user based on the frame difference information, including: Based on the frame difference information between each pair of adjacent image frames, find the two rows of pixels with the smallest sum of pixel differences among all pixels in the same row in the mixed video stream; Determine whether the two rows of pixels are sequential in position. If the two rows of pixels are sequential, use the positions of the two rows of pixels to determine the boundary information. When the boundary information is not determined, the boundary information is determined iteratively using the frame difference information between the next pair of adjacent image frames.

4. The video display method for live streaming according to claim 3, characterized in that, When two rows of pixels are consecutive, the boundary information is determined using the positions of these two rows of pixels, including: When two rows of pixels are consecutive, the positions of the two rows of pixels in the current judgment are compared with the positions of the two rows of pixels in the previous round. If the positions remain unchanged, the boundary information is determined using the positions of the two rows of pixels. Otherwise, the boundary information is determined iteratively using the frame difference information between the next pair of adjacent image frames.

5. The video display method for live streaming according to claim 1, characterized in that, Identify and extract the video frame size occupied by each broadcaster's video image in the full frame of the mixed video stream, including: Aspect ratio information is extracted from the supplementary enhancement information of the mixed video stream, and the aspect ratio information includes the proportion of the video image of each anchor user in the mixed video stream; Based on the full frame of the mixed video stream, the video frame occupied by the video image of each broadcaster user is calculated and determined according to the frame ratio information.

6. The method for displaying live video chat according to any one of claims 1 to 5, characterized in that, Displaying the video images of each broadcaster user in their respective target playback windows includes: Calculate the display resolution of the target playback window for each broadcaster's video image; Determine whether the display resolution of each video image is lower than a first resolution threshold but higher than a second resolution threshold. If the determination is true, use interpolation to perform pixel interpolation on the corresponding video image to achieve image quality enhancement. Determine whether the display resolution of each video image is lower than a second resolution threshold. If it is lower than the second resolution threshold, use a preset super-resolution enhancement model to enhance the image quality of the corresponding video image. The enhanced video image is then drawn and displayed in its corresponding target playback window.

7. A video display device for live streaming, characterized in that, include: The video streaming module is configured to acquire mixed video streams that are forwarded by the content distribution network during live streaming, wherein the mixed video streams contain video images of two or more live streaming users. The switching control module is configured to respond to the video layout switching command of the audience user and determine the target playback window of each anchor user as defined by the video layout switching command, wherein at least two of the target playback windows have a stacking relationship. The image extraction module is configured to identify and extract the video frame area occupied by each anchor user's video image in the full frame of the mixed video stream, and extract the video image of each anchor user according to each video frame area. The image display module is configured to draw and display the video images of each broadcaster user on the corresponding target playback window of each broadcaster user. The image frame calling module is configured to retrieve image frames from the mixed video stream; The special effects image segmentation module is configured to segment the image frame into special effects images using a preset image segmentation model; The special effects type determination module is configured to use a preset special effects classification model to determine the special effects type to which the special effects image belongs; The special effects local playback module is configured to execute the local special effects playback instruction corresponding to the special effects type, play the corresponding animation special effects synchronously with the image frame in the current graphical user interface, and cover the special effects image appearing in the target playback window with the largest display area with the image in the animation special effects.

8. The video display device for live streaming according to claim 7, characterized in that, The image display module includes: The image quality analysis unit is configured to calculate the display resolution of the video image of each broadcaster user corresponding to its target playback window; The interpolation enhancement unit is configured to determine whether the display resolution of each video image is lower than a first resolution threshold but higher than a second resolution threshold. When the determination is true, the corresponding video image is pixel interpolated using an interpolation method to achieve image quality enhancement. The depth enhancement unit is configured to determine whether the display resolution of each video image is lower than a second resolution threshold. When it is lower than the second resolution threshold, a preset super-resolution enhancement model is used to enhance the image quality of the corresponding video image. The display processing unit is configured to draw and display the enhanced video image in its corresponding target playback window.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

Images