Video microphone connection method and electronic equipment

Abstract

The invention provides a video microphone connection method and electronic equipment. The video microphone connection method comprises the following steps: acquiring a first picture, wherein the first picture comprises a picture acquired by a first terminal; a second picture is obtained, the second picture comes from a second terminal, and the second terminal is a terminal performing video microphone connection with the first terminal; extracting a figure layer of the second picture; and superposing the background layer of the first picture, the figure layer of the first picture and the figure layer of the second picture to obtain a first target picture displayed on the audience terminal device, the background layer of the first picture in the first target picture being located below the figure layer of the first picture and the figure layer of the second picture. According to the method, the character images of the microphone connection users performing video microphone connection based on the first terminal and the second terminal share one background image layer, so that the experience of audiences when watching the video microphone connection on audience terminal equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of live broadcast technology, and in particular to a video connection method and an electronic device. Background Art

[0002] Live broadcast technology is in a period of rapid development. Nowadays, in order to further improve the interactive experience, video linking can be carried out during the live broadcast. When video linking is carried out, the participating roles include the linking user (also called the linking anchor) and the audience.

[0003] At present, the live broadcast room picture seen by the audience on the audience-side device is formed by a simple rectangular combination and splicing of the images of all the connected anchors by the terminal used by the connected anchors or the live broadcast service platform. The image of each anchor seen by the audience is greatly reduced, resulting in a poor viewing experience for the audience when watching on the audience-side device. Summary of the invention

[0004] The present application provides a video connection method and an electronic device, in order to improve the viewing experience of viewers when watching a live broadcast screen during video connection on a viewer-side device.

[0005] In a first aspect, the present application provides a video connection method, including: obtaining a first picture, the first picture includes a picture collected by a first terminal, obtaining a second picture, the second picture comes from a second terminal, and the second terminal is a terminal for video connection with the first terminal; extracting a character layer of the second picture; superimposing a background layer of the first picture, a character layer of the first picture, and a character layer of the second picture to obtain a first target picture, wherein the character layer of the first picture and the character layer of the second picture are both superimposed under the background layer of the first picture.

[0006] The method may be executed by the first terminal.

[0007] In this embodiment, the first terminal and the second terminal perform a video connection. Specifically, from the user's perspective, the first terminal and the second terminal perform a video connection can also be expressed as: the first user and the second user perform a video connection, wherein the first user performs a video connection with the second user based on the first terminal, and the second user performs a video connection with the first user based on the second terminal.

[0008] Specifically, in this embodiment, the first picture refers to the picture captured by the camera of the first terminal when the first user uses the first terminal to conduct a video call with the second user who uses the second terminal. Similarly, the second picture refers to the picture captured by the camera of the second terminal when the second user uses the second terminal to conduct a video call with the first user who uses the first terminal.

[0009] In this embodiment, after the first terminal obtains the first picture and the second picture, the first picture and the second picture are processed as follows: the background layer and the character layer are extracted from the first picture, and the character layer is extracted from the second picture; the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture are superimposed (also called superimposed synthesis). In this embodiment, during the superimposed synthesis, the background layer of the first picture is placed below the character layer of the first picture and the character layer of the second picture to ensure that the character layer covers the background layer instead of the background layer covering the character layer.

[0010] In this embodiment, the picture obtained by superimposing and synthesizing the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture is called the first target picture. Preferably, the picture of the overlapping area obtained by superimposing and synthesizing the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture can be determined as the first target picture. It should be understood that the first target picture is the picture that the audience can see on the audience-side device (that is, the picture of the live broadcast room displayed when the first user and the second user are connected by video).

[0011] When ensuring that the background layer of the first screen is located below the character layer of the first screen and the character layer of the second screen, the character layer of the first screen and the character layer of the second screen can be superimposed based on the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen.

[0012] For example, if the ratio between the width of the forehead of the first user and the width of the forehead of the second user is greater than or equal to the first threshold, it is determined to overlay the background layer of the first screen, the character layer of the second screen, and the character layer of the first screen in the order of bottom to top. That is, when overlaying, the character layer of the first screen covers the character layer of the second screen.

[0013] For example, if the ratio between the width of the forehead of the first user and the width of the forehead of the second user is less than the first threshold, and the character layer of the second screen is allowed to be on top of the character layer of the first screen, it is determined to overlay the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen in a bottom-to-top order, that is, the character layer of the second screen is allowed to cover the character layer of the first screen during overlay.

[0014] For example, if the ratio of the width of the forehead of the first user to the width of the forehead of the second user is less than the first threshold, and the character layer of the second screen is not allowed to be on top of the character layer of the first screen, the background layer of the first screen and the character layer of the first screen are superimposed in the order from bottom to top. That is, the character layer of the second screen is directly discarded during superposition.

[0015] It should be understood that in the video connection method provided by the present application, since the character layer and the background layer are extracted from the first picture captured by the first terminal, and only the character layer is extracted from the second picture captured by the second terminal, and the three layers are superimposed and synthesized, the character images of the first user and the second user can share a background layer, so that the audience has the experience of the first user and the second user in the same physical space when watching, thereby improving the audience's viewing experience when watching the video connection between the first user and the second user on the audience-end device.

[0016] In combination with the first aspect, in a possible implementation method, the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture are superimposed to obtain a first target picture, including: superimposing the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture based on superposition parameters to obtain the first target picture, and the superposition parameters include at least one of the following parameters: a parameter indicating the scaling ratio between the character layer of the second picture and the character layer of the first picture, and a parameter indicating the height ratio between the character heights in the character layer of the second picture and the character heights in the character layer of the first picture.

[0017] That is, in this implementation, when the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are superimposed, the superposition is performed based on the superposition parameter. Specifically, during implementation, the character layer of the layer of the second screen is adjusted by the parameter indicating the zoom ratio and / or the parameter indicating the height ratio, so that the character layer of the second screen and the character layer of the first screen meet the above-mentioned zoom ratio and height ratio.

[0018] In combination with the first aspect, in an implementation manner, the method also includes: the first terminal sends a microphone connection request message, the microphone connection request message is used to request a video microphone connection with the second terminal; in response to a confirmation message that the second terminal agrees to the video microphone connection, an automatic negotiation interface is output, the automatic negotiation interface includes a second target screen and a first control, the second target screen is a screen of an overlapping area obtained by superimposing the background layer of the third screen, the character layer of the fourth screen and the character layer of the third screen in a bottom-to-top order based on set parameters, the second target screen includes characters in the character layer of the third screen and characters in the character layer of the fourth screen, and the characters in the character layer of the third screen included in the second target screen partially overlap with the characters in the character layer of the fourth screen; in response to an operation of clicking the first control, a second parameter is determined as the superposition parameter; wherein the second target screen can be obtained after the background layer of the third screen, the character layer of the fourth screen and the character layer of the third screen are superimposed in a bottom-to-top order using the second parameter.

[0019] Exemplarily, the third picture is a picture captured by the first terminal when a first user using the first terminal places his head in a designated area of ​​the left half screen of the first terminal, and the fourth picture is a picture captured by the second terminal when a second user using the second terminal places his head in a designated area of ​​the right half screen of the second terminal.

[0020] For example, the shoulders or arms of the characters in the character layer of the third screen included in the second target screen overlap with those in the character layer of the fourth screen, or the left leg of the character in the character layer of the third screen partially overlaps with the right leg of the character in the character layer of the fourth screen, etc., which does not constitute a limitation.

[0021] In this implementation, parameters (also called system-set parameters, second parameters) are set in the first terminal. Then, when the second terminal agrees to conduct a video link with the first terminal, the first terminal will overlay the background layer of the third screen, the character layer of the fourth screen, and the character layer of the third screen in order from bottom to top, and overlay based on the set parameters. That is, when the first terminal overlays the background layer of the third screen, the character layer of the fourth screen, and the character layer of the third screen in order from bottom to top, it will adjust the character layer of the fourth screen based on the set parameters so that the character layer of the fourth screen and the character layer of the third screen meet the scaling ratio and height ratio, and then overlay the background layer of the third screen and the character layer of the third screen and the character layer of the fourth screen that meet the scaling ratio and height ratio.

[0022] Exemplarily, the set parameters include at least one of the following parameters: a ratio parameter for the scaling ratio between the character layer of the fourth screen and the character layer of the third screen, a position parameter for the height ratio between the heights of the characters in the character layer of the fourth screen and the heights of the characters in the character layer of the third screen; in response to the operation of clicking the first control, the set parameters are determined as overlay parameters.

[0023] It should be noted that in this application, when the second terminal agrees to conduct a video connection, the second terminal will output the same automatic negotiation interface as the first terminal. In other words, when the second terminal agrees to conduct a video connection, the first terminal and the second terminal will output the same automatic negotiation interface.

[0024] In combination with the first aspect, in a possible implementation, the automatic negotiation interface also includes a second control, and the method also includes: in response to an operation of clicking the second control, displaying a manual negotiation interface, the manual negotiation interface including a second target screen; in response to adjusting the character layer of the fourth screen in the second target screen on the manual negotiation interface, displaying the adjusted third target screen; when the first user and the second user are satisfied with the third target screen, determining the first parameter as the overlay parameter; wherein, the third target screen can be obtained when the character layer of the fourth screen in the second target screen is adjusted using the first parameter.

[0025] The second control is also called a manual negotiation control.

[0026] In this embodiment, when the first user and the second user are not satisfied with the second target screen output by the automatic negotiation interface, it is allowed to trigger the output of the manual negotiation interface by clicking the second control, and then the first user is allowed to adjust the character layer of the fourth screen in the manual negotiation interface, for example, by adjusting the mouse or by touching the screen, so as to obtain the third target screen that the first user and the second user are satisfied with. Afterwards, when the first terminal superimposes the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen, it superimposes based on the first parameter. Among them, the third target screen can be obtained when the character layer of the fourth screen in the second target screen is adjusted using the first parameter.

[0027] In this embodiment, the second terminal and the first terminal both output the same manual negotiation interface.

[0028] In some embodiments, the second user may click the second control on the automatic negotiation interface on the second terminal to trigger the first terminal and the second terminal to output the manual negotiation interface.

[0029] In some embodiments, the first user may click the second control on the automatic negotiation interface on the first terminal to trigger the first terminal and the second terminal to output the manual negotiation interface.

[0030] In some embodiments, the manual negotiation interface includes a third control; determining that the first user and the second user are satisfied with the third target screen includes: if the first user clicks the third control on the first terminal and the second user clicks the third control in the manual negotiation interface on the second terminal, determining that the first user and the second user are satisfied with the third target screen.

[0031] In some embodiments, the first user and the second user may inform each other that they are satisfied with the third target image through voice feedback.

[0032] In combination with the first aspect, in a possible implementation method, when the background layer of the first picture, the character layer of the first picture and the character layer of the second picture are superimposed based on the overlay parameters, it includes: scaling the character layer of the second picture based on the parameter indicating the scaling ratio between the character layer of the second picture and the character layer of the first picture; aligning the center point of the character layer of the first picture with the center point of the scaled character layer of the second picture; and moving the scaled character layer of the second picture up and down based on the parameter indicating the height ratio between the character layer of the second picture and the character height in the character layer of the first picture, so that the height ratio between the character layer of the second picture and the character height in the character layer of the first picture is satisfied.

[0033] In a second aspect, the present application provides a video connection method, which is applied to a live broadcast service platform; comprising: obtaining a first picture, the first picture includes a picture collected by a first terminal; obtaining a second picture, the second picture comes from a second terminal, and the second terminal is a terminal that conducts a video connection with the first terminal; extracting a background layer of the first picture, a character layer of the first picture, and a character layer of the second picture; superimposing the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture to obtain a first target picture, wherein the background layer of the first picture in the first target picture is located below the character layer of the first picture and the character layer of the second picture.

[0034] Among them, the detailed description of the second aspect can refer to the detailed description of the first aspect, and the only difference is that the video connection method is executed by the live broadcast service platform, which will not be repeated here.

[0035] In conjunction with the second aspect, in a possible implementation manner, the first target picture is a picture of an overlapping area after the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture are superimposed.

[0036] In combination with the second aspect, in a possible implementation method, the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are superimposed, including: obtaining the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen; if the ratio between the forehead width of the first user and the forehead width of the second user is greater than or equal to a first threshold, superimposing the background layer of the first screen, the character layer of the second screen, and the character layer of the first screen in order from bottom to top.

[0037] In combination with the second aspect, in a possible implementation method, superimposing the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen includes: obtaining the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen; if the ratio between the forehead width of the first user and the forehead width of the second user is less than a first threshold, and the character layer of the second screen is allowed to be above the character layer of the first screen, determining to superimpose the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen in a bottom-to-top order.

[0038] In combination with the second aspect, in a possible implementation method, the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are superimposed, including: obtaining the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen; if the ratio of the forehead width of the first user to the forehead width of the second user is less than a first threshold, and the character layer of the second screen is not allowed to be above the character layer of the first screen, superimposing the background layer of the first screen and the character layer of the first screen in order from bottom to top.

[0039] In combination with the second aspect, in a possible implementation method, when the background layer of the first picture, the character layer of the first picture and the character layer of the second picture are superimposed to obtain the first target picture, it also includes: superimposing the background layer of the first picture, the character layer of the first picture and the character layer of the second picture based on the superposition parameter to obtain the first target picture, and the superposition parameter includes at least one of the following parameters: a parameter indicating the scaling ratio between the character layer of the second picture and the character layer of the first picture, and a parameter indicating the height ratio between the character heights in the character layer of the second picture and the character heights in the character layer of the first picture.

[0040] In a third aspect, the present application provides a video connection device, including a functional module for implementing the method described in the first aspect or any possible implementation manner thereof.

[0041] In a fourth aspect, the present application provides a video connection device, comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to execute the program instructions in the memory to implement the method described in the first aspect or any possible implementation method thereof.

[0042] In a fifth aspect, the present application provides an electronic device, including the third aspect or the video connection device described in the third aspect.

[0043] In a sixth aspect, the present application provides a video connection device, including a functional module for implementing the method described in the second aspect or any possible implementation method thereof.

[0044] In a seventh aspect, the present application provides a live broadcast server, comprising the video connection device described in the sixth aspect.

[0045] In an eighth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code for computer execution, wherein the program code includes instructions for implementing the method described in the first aspect, the second aspect, or any possible implementation method thereof.

[0046] In a ninth aspect, the present application provides a computer program product, the computer program product comprising instructions for implementing the method as described in the first aspect or any possible implementation method thereof.

[0047] It can be understood that the effects that can be obtained from the second to ninth aspects can be referred to the description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in this application;

[0049] Figure 2 A schematic diagram of a software structure of an electronic device provided for this application;

[0050] Figure 3 A schematic diagram of the video link system provided in this application;

[0051] Figure 4 A schematic diagram of the live broadcast room screen displayed on the viewer side when the existing viewers watch the video with microphone;

[0052] Figure 5 A schematic diagram of the process of the video linking method provided in this application;

[0053] Figure 6 A schematic diagram of determining the occlusion relationship between the character layer of the first picture and the character layer of the second picture provided by the present application;

[0054] Figure 7 A schematic diagram of the structure of a video microphone connection system provided by the present application and applicable to the video microphone connection method of the present application;

[0055] Figure 8 A schematic diagram of another video linking method provided in this application;

[0056] Fig. 9 A schematic diagram of the host host end inviting the guest host end to join the live broadcast room and the guest host end responding to the invitation provided by this application;

[0057] Fig.10A schematic diagram of the automatic negotiation interface provided for this application;

[0058] Fig.11 Schematic diagram of the three extracted layers;

[0059] Fig.12 A schematic diagram of determining a scaling parameter according to an embodiment of the present application;

[0060] Fig.13 A schematic diagram showing the center position of the host character layer and the center position of the guest character layer;

[0061] Fig.14 A schematic diagram showing adjusting the height of characters in a guest character layer is shown;

[0062] Fig.15 A schematic diagram showing adjustment triggering manual negotiation to display a manual negotiation interface is shown;

[0063] Fig.16 A structural schematic diagram of a video link device provided in one embodiment of the present application;

[0064] Fig.17 A structural schematic diagram of a video link device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0066] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0067] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0068] In addition, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or multiple.

[0069] Before introducing the video linking method of the present application, in order to better understand the embodiments of the present application, the electronic device of the embodiments of the present application is introduced below.

[0070] For example, Figure 1 A schematic diagram of the structure of an electronic device (also called a terminal device) provided in an embodiment of the present application.

[0071] Figure 1 The terminal device may include a processor 110, an internal memory 121, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a sensor module 180, a button 190, a display screen 194, a timer 196, and the like.

[0072] Optionally, the sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, and the like.

[0073] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, etc. Among them, different processing units may be independent devices or integrated into one or more processors. A memory may also be provided in the processor 110 for storing instructions and data.

[0075] The charging management module 140 is used to receive charging input from a charger, which may be a wireless charger or a wired charger. The power management module 141 is used to connect the charging management module 140 to the processor 110 .

[0076] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0077] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna.

[0078] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to terminal devices. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.

[0079] The wireless communication module 160 can provide wireless communication solutions applied to terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc.

[0080] In some embodiments, antenna 1 of the terminal device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that the mobile phone can communicate with the network and other devices through wireless communication technology.

[0081] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0082] The terminal device can realize the display function through a graphics processing unit (GPU), a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The GPU can also be called a display core, a visual processor, a display chip, etc.

[0083] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store applications required for at least one function in the operating system (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the terminal device (such as audio data, a phone book, etc.).

[0084] The terminal device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, and the application processor.

[0085] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The terminal device can listen to music or listen to hands-free calls through the speaker 170A. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device answers a call or voice message, the voice can be answered by placing the receiver 170B close to the human ear. The microphone 170C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0086] The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be arranged on the display screen 194. When a touch operation acts on the display screen 194, the terminal device can detect the intensity of the touch operation according to the pressure sensor 180A. The terminal device can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. The touch sensor 180B can be arranged on the display screen 194, and the touch sensor 180B and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor 180B can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180B can also be arranged on the surface of the mobile phone, which is different from the position of the display screen 194.

[0087] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button. It may also be a touch button. The terminal device may receive the button input and generate a key signal input related to the user settings and function control of the terminal device. The indicator 192 may be an indicator light, which may be used to indicate the charging status, the change of the power, or may be used to indicate messages, missed calls, notifications, etc. The timer 196 may be used to record time information. For example, when the terminal device records a video, the start time and the end time of the video may be recorded, and the duration of the recorded video may be recorded.

[0088] Figure 2 A software structure block diagram of a terminal device applicable to the embodiments of the present application. The layered architecture divides the software system of the terminal device into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers, from top to bottom, namely, the application layer (applications), the application framework layer (application framework), the Android runtime (Android runtime) and the system library, and the kernel layer (kernel).

[0089] The application layer may include a series of application packages, and the application layer runs applications by calling the application programming interface (API) provided by the application framework layer.

[0090] For example, an application package may include desktop, calendar, memo, map, navigation, Bluetooth, music, JD.com, and short message applications.

[0091] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0092] For example, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0093] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, the display interface including the SMS notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for terminal devices. For example, the management of call status (including connected, hung up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of an icon or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0094] The Android system runtime includes the core library and the virtual machine. The Android system runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection. The system library can contain modules with multiple functions, such as: surface manager, media library, and 3D graphics processing library.

[0095] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.

[0096] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make the hardware work. The kernel layer at least includes display driver, screen driver, camera driver and sensor driver, etc., which are not limited in the embodiments of the present application.

[0097] Live broadcast technology is booming. In the Internet live broadcast business, the initial business form was a one-way video display, that is, the audience could only watch the live broadcast of the anchor in one direction. Later, it gradually developed into the audience being able to interact with the anchor through text and bullet screen. Nowadays, in order to further improve the interactive experience, video connection can be carried out during the live broadcast. For example, when a user Figure 1 or Figure 2 The electronic device shown can connect with other users via video when broadcasting live.

[0098] During video connection, the participants include the users (at least two) and the audience. After the users successfully connect, the audience can see the images of all the users on the audience side device at the same time.

[0099] In this application, users participating in video conferencing are also referred to as conferencing hosts.

[0100] For example, Figure 3 A schematic diagram of a video link system is given. Figure 3 The system includes terminal 1, terminal 2, live broadcast service platform 3 (also called live broadcast server, live broadcast server platform, live broadcast platform, etc., which are not limited), and terminal 4. Among them, anchor A can use terminal 1 to connect with anchor B who uses terminal 2 based on live broadcast service platform 3. Correspondingly, the live broadcast service platform pushes the live broadcast room screen when anchor A and anchor B connect to the microphone to terminal 4.

[0101] In some embodiments, terminal 1 is the end that initiates the video connection, and terminal 2 is the end that receives the video connection. In this case, terminal 1 can be called the video connection initiator, and terminal 2 can be called the video connection receiver.

[0102] In some embodiments, terminal 2 is the end that initiates the video connection, and terminal 1 is the end that receives the video connection. In this case, terminal 2 can be called the video connection initiator, and terminal 1 can be called the video connection receiver.

[0103] It is to be noted here that Figure 3 The system architecture of is only illustrated by using two anchors to conduct video linking, but it does not constitute a limitation of the present application. For example, the system architecture may also include terminal 4 and terminal 5, anchor C may use terminal 3 to conduct video linking with anchor D who uses terminal 4 based on live broadcast service platform 3. Or, it may also include terminal 6, anchor A uses terminal 1 to conduct video linking with anchor B who uses terminal 2, and also conducts video linking with anchor E who uses terminal 6 through live broadcast service platform 3.

[0104] In this application, the terminal used for video conferencing is also referred to as the host terminal, and the terminal used by the audience watching the video conferencing is also referred to as the audience terminal or the audience side device. Figure 3 In the example, terminal 1 and terminal 2 are both anchor terminals, and terminal 4 is a viewer-side device.

[0105] Specifically, the live broadcast room picture currently seen by the audience on the audience-side device is formed by the anchor terminal or the live broadcast service platform simply combining and stitching the images of all the connected anchors into a rectangular shape.

[0106] For example, taking a common one-on-one live broadcast as an example, Figure 4 This is a schematic diagram of the live broadcast room when the video played on the audience side device is connected to the microphone. Figure 4 As shown, the audience-side device includes a display interface 400. The left side of the display interface 400 is a picture including anchor A collected based on the terminal used by anchor A, and the right side of the display interface 400 is a picture including anchor B collected based on the terminal used by anchor B. The two pictures displayed on the audience-side device are spliced ​​in parallel from left to right.

[0107] However, this simple splicing method using rectangular combinations results in a significant reduction in the image of the host viewed by the audience. For example, in a one-on-one video connection scenario, the image of a single host is reduced to 25%. The image of a single host cannot be presented in detail, resulting in a poor viewing experience for the audience.

[0108] In view of this, the present application provides a method for video conferencing, in order to improve the viewing experience of viewers during video conferencing.

[0109] The following describes a first embodiment of the video linking method provided by the present application in conjunction with the accompanying drawings.

[0110] The video linking method provided in this embodiment is executed by a terminal that initiates the video linking, and the terminal is referred to as a first terminal; accordingly, a terminal that responds to the first terminal that initiates the video linking and connects the video linking with the first terminal is referred to as a second terminal. Figure 5 As shown, the method includes:

[0111] S501, a first terminal obtains a first picture and a second picture, where the first picture includes a picture collected by the first terminal, and the second picture comes from a second terminal, which is a terminal connected to the first terminal via video.

[0112] From the user's perspective, the video call between the first terminal and the second terminal can also be expressed as: the first user and the second user have a video call, wherein the first user is a user who has a video call with the second user based on the first terminal, and the second user is a user who has a video call with the first user based on the second terminal.

[0113] In this embodiment, the first user is also referred to as a first anchor, a host anchor, etc., and the second user is also referred to as a second anchor, a guest anchor, etc.

[0114] In this embodiment, the picture captured by the camera of the first terminal when the first user and the second user are connected via video is called the first picture (that is, the image picture of the physical space where the first user is located captured by the camera of the first terminal), and the picture captured by the camera of the second terminal when the second user and the first user are connected via video is called the second picture (that is, the image picture of the physical space where the second user is located captured by the camera of the second terminal).

[0115] It should be noted that this embodiment does not limit how the first terminal obtains the second picture captured by the camera of the second terminal: for example, in one implementation, the second terminal can directly send the captured second picture to the first terminal. For another example, in another implementation, the live broadcast service platform can forward the second picture captured by the camera of the second terminal to the first terminal.

[0116] S502: The first terminal extracts a background layer of a first picture, a character layer of the first picture, and a character layer of a second picture.

[0117] In this embodiment, after the first terminal obtains the first picture and the second picture, it will extract the background layer of the first picture and the character layer of the first picture, and extract the character layer of the second picture. The concepts of layer, background layer, and character layer can refer to the description in the relevant technology, which will not be repeated here.

[0118] S503, the first terminal superimposes the background layer of the first picture, the character layer of the first picture and the character layer of the second picture to obtain a first target picture, wherein the character layer of the first picture and the character layer of the second picture are both superimposed under the background layer of the first picture.

[0119] In this embodiment, after the first terminal extracts the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen, the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are re-overlaid. In this embodiment, when overlaying, the background layer of the first screen is placed below the character layer of the first screen and the character layer of the second screen to ensure that the character layer covers the background layer instead of the background layer covering the character layer.

[0120] In this embodiment, after the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are superimposed, the synthesized screen is called the first target screen. Preferably, the screen of the overlapping area after the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are superimposed can be determined as the first target screen. Among them, the first target screen is the screen that the audience can see on the audience-side device (that is, the screen of the live broadcast room displayed when the first user and the second user are connected by video).

[0121] In some embodiments, on the basis of ensuring that the background layer of the first screen is located below the character layer of the first screen and the character layer of the second screen, when superimposing, the first terminal can determine the superimposition order of the character layer of the first screen and the character layer of the second screen based on the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen.

[0122] Exemplarily, an example of determining the stacking order of the character layer of the first screen and the character layer of the second screen based on the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen is as follows: Figure 6 As shown, including:

[0123] S601, collecting the forehead width of a first user in a character layer of a first picture and the forehead width of a second user in a character layer of a second picture.

[0124] For example, the data is collected with a period of 1 second.

[0125] S602, determine whether the ratio between the forehead width of the first user and the forehead width of the second user is greater than or equal to a first threshold, if so, execute S603, otherwise, execute S604.

[0126] S603: Determine to overlay the background layer of the first picture, the character layer of the second picture, and the character layer of the first picture in order from bottom to top.

[0127] In other words, the character layer of the first screen is made to cover the character layer of the second screen when superimposed; or in other words, it is determined that the covering relationship between the first user and the second user when superimposed is that the first user covers the second user.

[0128] S604, determining whether the first terminal is set to allow the character layer of the second screen to be above the character layer of the first screen. If so, executing S605, otherwise, executing S606.

[0129] Among them, determining whether the first terminal is set to allow the character layer of the second screen to be above the character layer of the first screen can also be described as: determining whether the first terminal is set to allow the second user to block the first user.

[0130] S605: Determine to overlay the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture in order from bottom to top.

[0131] In other words, the character layer of the second screen covers the character layer of the first screen when superimposed; or in other words, the covering relationship between the first user and the second user when superimposed is determined to be that the second user covers the first user.

[0132] S606: Determine to overlay the background layer of the first picture and the character layer of the first picture in order from bottom to top.

[0133] In other words, the character layer of the second picture is cancelled during superposition. Optionally, in this case, the second terminal may output prompt information to prompt the second user to adjust the position.

[0134] As an example, if the first user and the second user have the same gender, when the ratio between the forehead width of the first user and the forehead width of the second user is greater than or equal to 1:1*(1+10%), the occlusion relationship is determined to be that the first user occludes the second user; and when the ratio between the forehead width of the first user and the forehead width of the second user is less than 1:1*(1+10%), for example, the second user is too close to the camera of the second terminal due to movement or other reasons, resulting in the ratio between the forehead width of the first user and the forehead width of the second user being less than 1:1*(1+10%), at this time, if the first terminal is set to allow the second user to occlude the first user, the occlusion relationship is determined to be that the second user occludes the first user, and if the first terminal is set not to allow the second user to occlude the first user, the character layer of the second screen is canceled when superimposing (that is, the background layer of the first screen and the character layer of the first screen are superimposed from bottom to top). It should be understood that in this example, 1:1*(1+10%) can be considered as the first threshold. The above 10% can be set in advance, also known as a set deviation threshold.

[0135] It should be understood that the above-mentioned forehead width is only an example and can be replaced by other parts, which does not constitute a limitation of the present application.

[0136] For example, in other embodiments, the occlusion relationship between the first user and the second user can also be determined by the binaural distance or the distance from the chin to the top of the head. Taking the binaural distance as an example, a second threshold can be set, and then the occlusion relationship between the first user and the second user is determined by judging the ratio of the binaural distance of the first user to the binaural distance of the second user and the second threshold. Taking the distance from the chin to the top of the head as an example, a third threshold can be set, and then the occlusion relationship between the first user and the second user is determined by judging the ratio of the distance from the chin to the top of the head of the first user to the distance from the chin to the top of the head of the second user and the third threshold.

[0137] For another example, in some other embodiments, considering the head movements of the first user and the second user, if the occlusion relationship between the first user and the second user is determined only by the forehead width alone, or the interaural distance alone, or the distance from the chin to the top of the head alone, then inaccurate judgment may occur. Therefore, in another example, a joint judgment can be performed. For example, the forehead width, interaural distance, and the distance from the chin to the top of the head are collected at the same time, and then the ratio between the forehead widths of the first user and the second user, the ratio between the interaural distances, and the ratio between the distance from the chin to the top of the head are calculated, and then the ratio with the smallest ratio is taken as the effective ratio, and the occlusion relationship between the first user and the second user is determined based on the effective ratio. For example, if the effective ratio is the ratio between the interaural distances, the occlusion relationship between the first user and the second user is determined by the size relationship between the effective ratio and the second threshold.

[0138] It can be seen that in the video connection method provided by the present application, since the character layer and the background layer are extracted from the first picture collected by the first terminal, and only the character layer is extracted from the second picture collected by the second terminal, and then the three layers are superimposed and synthesized, since there is only one background layer (that is, it is integrated into one background or unified), from the audience's perspective, the audience thinks that the first user and the second user are in the same physical space when watching, providing a viewing experience of multiple connected hosts interacting in the same live studio environment. In addition, in the video connection method, the character image of each connected user no longer needs to be greatly reduced, so that the audience can still observe the details of the connected user, thereby improving the audience's viewing experience when watching the video connection between the first user and the second user on the audience-side device.

[0139] From the perspective of the users of the video conferencing, since this method places the hosts of the video conferencing on the same background, the hosts of the video conferencing can carry out some interactive interactions, making the video conferencing richer.

[0140] Furthermore, in the present application, when the first terminal superimposes the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen to obtain the first target screen, the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen are superimposed by superposition parameters to obtain the first target screen.

[0141] Specifically, the overlay parameter is used to indicate how to adjust the character layer of the second screen during overlay, so that the character layer of the second screen satisfies a certain relationship with the character layer of the first screen, so that the synthesized screen presents a better effect. Exemplarily, the overlay parameter includes at least one of the following parameters: a parameter indicating the scaling ratio between the character layer of the second screen and the character layer of the first screen (also referred to as a scaling ratio parameter in this application), a parameter indicating the height ratio between the character layer of the second screen and the character height in the character layer of the first screen (also referred to as a height ratio parameter or a position parameter or a height position parameter in this application).

[0142] That is to say, in this embodiment, before superimposing three layers to obtain the first target picture, the first terminal also needs to adjust the character layer of the second picture through the zoom ratio parameter and / or height ratio parameter so that the zoom ratio and height ratio are satisfied between the character layer of the second picture and the character layer of the first picture.

[0143] In some embodiments, the superposition parameter is a second parameter set in advance in the first terminal. For example, during implementation, after the second user agrees to conduct a video link with the first user, the first terminal may first output an automatic negotiation interface to the first user, and on the other hand, the second terminal also outputs an automatic negotiation interface to the second user, and the automatic negotiation interface output by the first terminal and the second terminal includes a second target screen and a first control. Among them, the second target screen is a screen of the overlapping area obtained by the first terminal using the second parameter to superimpose the background layer of the third screen, the character layer of the fourth screen, and the character layer of the third screen in a bottom-to-top order, and the second target screen includes the characters in the character layer of the third screen and the characters in the character layer of the fourth screen, and the characters in the character layer of the third screen included in the second target screen partially overlap with the characters in the character layer of the fourth screen; thereafter, when the first user and the second user click the first control, the first terminal determines the second parameter as the superposition parameter used for the subsequent superposition of the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen. Or in other words, when the first user and the second user click on the first control, it indicates that the first user and the second user are relatively satisfied with the picture of the overlapping area obtained after the first terminal uses the second parameter to superimpose the background layer of the third screen, the character layer of the fourth screen and the character layer of the third screen in order from bottom to top, so that the first terminal determines the second parameter as the superposition parameter used for subsequent superposition of the background layer of the first screen, the character layer of the first screen and the character layer of the second screen.

[0144] For example, the shoulders or arms of the characters in the character layer of the third screen included in the second target screen overlap with those in the character layer of the fourth screen, or the left leg of the character in the character layer of the third screen partially overlaps with the right leg of the character in the character layer of the fourth screen, etc., which does not constitute a limitation.

[0145] Furthermore, in some other embodiments, a second control may be included on the automatic negotiation interface. If the first user and / or the second user clicks the second control, the first terminal and the second terminal display a manual negotiation interface, which includes the second target screen; thereafter, in response to the first user adjusting the character layer of the fourth screen in the second target screen on the manual negotiation interface, the first terminal and the second terminal respectively display the adjusted third target screen, and when the first user and the second user are satisfied with the third target screen, the first parameter is determined as the superposition parameter; wherein, the third target screen can be obtained when the character layer of the fourth screen in the second target screen is adjusted using the first parameter.

[0146] Exemplarily, the third picture is a picture captured by the first terminal when a first user using the first terminal places his head in a designated area of ​​the left half screen of the first terminal, and the fourth picture is a picture captured by the second terminal when a second user using the second terminal places his head in a designated area of ​​the right half screen of the second terminal.

[0147] Below, for ease of understanding, taking the third picture being the picture captured by the first terminal when the first user using the first terminal places the head in the designated area of ​​the left half screen of the first terminal, and the fourth picture being the picture captured by the second terminal when the second user using the second terminal places the head in the designated area of ​​the right half screen of the second terminal as examples, a second embodiment is given to explain in detail how to superimpose the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture based on the overlay parameters to synthesize the implementation process of the picture displayed on the audience-end device.

[0148] First, before introducing the second embodiment, a system architecture applied to the video linking method provided by the present application is introduced. Figure 5 The system architecture for video linking provided in this application includes a host end, a live broadcast service platform and an audience end. Specifically, the host end includes two parts: an end that initiates video linking and an end that receives video linking.

[0149] The end that initiates the video link is the corresponding Figure 5 The first terminal in the embodiment shown is the end receiving the video link. Figure 5 The second terminal in the illustrated embodiment.

[0150] For ease of description, in this embodiment, the end that initiates the video link is called the host host end, and the end that receives the video link is called the guest host end. The user who conducts the video link on the host host end is called the host host (i.e., the corresponding Figure 5 The first user described in the description), the user who conducts video connection on the guest host terminal is called the guest host (i.e. the corresponding Figure 5 ) described in the second user.

[0151] like Figure 7 As shown, the host anchor end includes a microphone connection management module 7011, a microphone connection receiving stream module 7012, a matte module 7013, a synthesis parameter control module 7014, a layer synthesis module 7015, a microphone connection sending stream module 7016 and a live streaming module 7017.

[0152] The microphone receiving stream module 7012 is used to receive the source stream sent by the microphone sending stream module of the guest host end (i.e., the picture stream collected by the guest host end), and send the source stream sent by the microphone sending stream module of the guest host end to the matte module 7013. It should be understood that the picture stream corresponds to a frame by frame.

[0153] The matting module 7013 is used to extract portraits based on portrait recognition technology and matting technology, extract the background layer and character layer of each frame captured by the host anchor end, and separately extract the character image of each frame captured by the guest anchor end.

[0154] In this embodiment, for the sake of convenience of description, the background layer extracted from the picture collected by the host anchor end is called the host background layer, the character layer extracted from the picture collected by the host anchor end is called the host character layer, and the character layer extracted from the picture collected by the guest anchor end is called the guest character layer.

[0155] The synthesis parameter control module 7014 is used to identify the forehead width and head position of both the host anchor and the guest anchor based on the portrait recognition technology, and determine the scaling parameters and / or height ratio parameters used to adjust the guest character layer, so that the host character layer and the guest character layer are arranged in a relatively reasonable / coordinated ratio and height position, so that the host anchor and the guest anchor visually appear to be arranged in a relatively reasonable ratio and height position. Specifically, in this embodiment, the synthesis parameter control module supports both automatic adjustment and manual adjustment. Once the scaling parameters and the height ratio parameters are determined, they are fixed throughout the live broadcast process. In addition, the synthesis parameter control module is also used to determine the occlusion relationship between the host character layer and the guest character layer.

[0156] The layer synthesis module 7015 is used to synthesize the live broadcast room screen stream (also referred to as synthesized stream or synthesized video stream in this application) after the video is connected to the microphone based on the layer synthesis technology according to the scaling ratio parameters and / or height ratio parameters sent by the synthesis parameter control module 7014, and the occlusion relationship between the host character layer and the guest character layer, send the synthesized stream to the microphone sending stream module 7016, and send the synthesized stream to the live broadcast push module 7017 to push it to the live broadcast service platform based on the live broadcast push module 7017.

[0157] The microphone sending stream module 7016 is used to collect the synthesized stream synthesized by the layer synthesis module 7015 and send the synthesized stream to the microphone receiving stream module of the guest anchor end.

[0158] like Figure 7As shown, the guest anchor terminal includes a microphone connection management module 7011', a microphone connection stream sending module 7016', a microphone connection stream receiving module 7012', a live streaming module 7017' and a microphone connection preview module 7018'.

[0159] The microphone sending stream module 7016' is used to send the images collected by the camera of the guest anchor end to the microphone receiving stream module of the host anchor end.

[0160] The microphone receiving stream module 7012' is used to receive the synthesized stream sent by the microphone sending stream module of the host anchor end, pass the synthesized stream to the microphone preview module 7018', and also send the synthesized stream to the live streaming push module 7017' to push it to the live streaming service platform based on the live streaming push module 7017'.

[0161] The microphone preview module 7018' is used to present the synthesized stream received by the microphone receiving stream module 7012' to the microphone host, so that the microphone host can choose manual negotiation and / or confirmation.

[0162] In this application, the microphone connection management module included in the host live broadcast end and the guest live broadcast end has added an "enter the live broadcast room" microphone connection mode, supports negotiation and control logic under the "enter the live broadcast room" microphone connection mode, and supports microphone connection management request interaction with the microphone connection management service module in the live broadcast service platform.

[0163] like Figure 7 As shown, the live broadcast service platform includes a live broadcast management service module and a live broadcast media module.

[0164] On the basis of maintaining the existing functions, the application adds the "enter the live broadcast room" live broadcast mode and the invitation and matching logic in the "enter the live broadcast room" live broadcast mode. For specific descriptions, please refer to the descriptions in the subsequent embodiments, which will not be repeated here.

[0165] Among them, "entering the live broadcast room" is also called joining the live broadcast room, which does not constitute a restriction.

[0166] The live media module is used by the audience to obtain the composite stream.

[0167] It is noted here that the live streaming module, live media module and audience end are common business modules in the live broadcast business. For a more detailed description, please refer to the description in the relevant technology, which will not be repeated here.

[0168] exist Figure 7 Based on the architecture shown, combined with Figure 8 , the second embodiment provided in this application is introduced in detail.

[0169] like Figure 8 As shown, the method includes:

[0170] S801, the host host terminal initiates a directed invitation request of "entering the live broadcast room" to the live broadcast service platform; correspondingly, the live broadcast service platform receives the directed invitation request.

[0171] Specifically, the directional invitation request includes a first identifier, and the first identifier indicates that the microphone connection mode of "entering the live broadcast room" provided in this application is adopted for microphone connection. That is, in this embodiment, the "entering the live broadcast room" microphone connection mode is newly added / expanded.

[0172] Specifically, the directional invitation request includes a second identifier, and the second identifier indicates the guest host to be invited for video linking. Exemplarily, the second identifier is the identifier of the guest host to be invited or the identifier of the live broadcast room when the guest host to be invited is broadcasting live.

[0173] It should be noted that this embodiment does not limit the specific form of the guest anchor. For example, in some embodiments, the guest anchor is selected by the host anchor on his own; in other embodiments, the guest anchor is automatically matched by the live broadcast service platform for the host anchor.

[0174] In practice, this targeted invitation request can be made by Figure 7 The connection is initiated by the microphone management module in the host anchor end, and then received by the microphone management service module in the live broadcast server end.

[0175] S802, the live broadcast service platform searches for the guest anchor who is connected to the host anchor according to the targeted invitation request and notifies the guest anchor of the invitation to "enter the live broadcast room".

[0176] For example, the notification includes at least an indication of "entering the live broadcast room" and a third identifier, wherein the third identifier indicates the host anchor. Exemplarily, the third identifier is the identifier of the host anchor or the identifier of the live broadcast room of the host anchor.

[0177] As an example, Fig. 9 A schematic diagram of a host host inviting a guest host to join a live broadcast room and the guest host responding to the invitation is provided in this application. Fig. 9 As shown, for the host host, the interface of the host host displays that it is inviting Y to join the live broadcast room. For the guest host, the interface of the guest host displays a notification sent by the live broadcast service platform, which includes "X invites you to join his live broadcast room, please choose whether to join", "Agree" control, "Reject" control and the image of the host host.

[0178] In practice, this notification may be Figure 7 The connection to the microphone management service module in the live broadcast service platform shown is initiated and received by the connection to the microphone management module in the guest anchor terminal.

[0179] S803, after the guest host receives the invitation to "enter the live broadcast room" and confirms that it agrees to enter, the guest host sends a notification message of agreement to the live broadcast service platform.

[0180] For example, the notification message includes at least a confirmation result field, which indicates that the guest host terminal agrees to have a video connection with the host host terminal.

[0181] In implementation, the notification message can be sent by Figure 7 The connection to the microphone management module in the guest host terminal shown is initiated, and correspondingly, it is received by the connection to the microphone management service module in the live broadcast service platform.

[0182] S804, after receiving the notification message of consent from the guest anchor, the live broadcast service platform sends a notification of successful microphone connection / notification of entering the live broadcast room to the host anchor.

[0183] For example, the notification message includes at least a guest host client confirmation result field to indicate to the host host that the guest host client agrees to the video link.

[0184] During implementation, the host can be notified of the successful connection to the live broadcast room or the guest's consent to enter the live broadcast room. Figure 7 The live broadcast service platform shown initiates the call through the microphone connection management service module, and accordingly, the call is received by the microphone connection management module of the host anchor terminal.

[0185] S805, the live broadcast service platform sends a notification of successful microphone connection / notification of entering the live broadcast room to the guest anchor.

[0186] During implementation, the notification of successful connection to the guest host / notification of entering the live broadcast room can be sent by Figure 7 The connection management service module of the live broadcast service platform shown is initiated, and correspondingly, it is received by the connection management module of the guest anchor end.

[0187] S806, the host host end divides the left half of the screen as the reference position of the host host, circles the area where the host host's head is recommended to be located with the left half of the screen as the center, and prompts the host host to look straight at the camera and adjust the position of the head; on the other hand, the guest host end divides the right half of the screen as the reference position of the guest host, circles the area where the guest host's head is recommended to be located with the right half of the screen as the center, and prompts the guest host to look straight at the camera and adjust the position of the head.

[0188] Among them, it is recommended that the area where the host's head is located can be Figure 7 The synthesis parameter control module in the host anchor terminal shown is provided, and the area where the guest anchor's head is located can be provided by the synthesis parameter control module in the guest anchor terminal.

[0189] For example, the synthesis parameter control module of the host anchor terminal may include a position parameter indicating the area where the head is located and a shape parameter indicating the shape of the area where the head is located.

[0190] As an example, Fig.10 The present application provides a user interface (also referred to as an automatic negotiation interface) for the host host end and the guest host end after receiving a notification of successful microphone connection / notification of entering the live broadcast room. Fig.10 As shown, the automatic negotiation interface includes manual negotiation controls, confirmation controls, relevant prompt information indicating the areas where the heads of the host anchor and the guest anchor should be located, and prompt information "Automatic Negotiation in Progress" indicating that the current interface is an automatic negotiation interface.

[0191] In the present application, when the host host places the head area of ​​the host host in the designated area through the automatic negotiation interface, the picture captured by the camera of the host host is the third picture.

[0192] In the present application, when the guest host places the head area of ​​the guest host in the designated area through the automatic negotiation interface, the picture captured by the camera of the guest host is the fourth picture.

[0193] S807, the guest host terminal sends the collected fourth picture to the host host terminal.

[0194] For example, based on RTC technology, the guest host establishes a connection with the host host through the RTC network, so that the fourth picture captured by the camera of the guest host when the guest host's head is in the recommended area is pushed to the host host through the RTC protocol.

[0195] In implementation, this step can be performed by Figure 7 The guest host terminal is executed by the microphone sending stream module shown; correspondingly, the host host terminal is executed by the microphone receiving stream module.

[0196] S808, the host host terminal calls the matting module to extract the host character layer and the host background layer based on the third picture collected by the host host terminal, and extracts the guest character layer from the fourth picture collected by the guest host terminal. Afterwards, the host host terminal calls the synthesis parameter control module to determine the scaling parameter so that the host host terminal adjusts the ratio between the guest character layer and the host character layer.

[0197] For example, Fig.11 The schematic diagram of the host character layer and background layer extracted by the matting module provided in this application from the third picture captured by the camera of the host anchor end, and the guest character layer extracted from the fourth picture received by the microphone receiving stream module.

[0198] Exemplarily, when the host anchor calls the synthesis parameter control module to determine the scaling parameters used in the automatic negotiation stage, a method is as follows: the synthesis parameter control module determines whether the genders of the host anchor and the guest anchor are the same based on face recognition technology. If they are the same, then it is determined to scale the guest character layer so that the ratio of the forehead width of the guest anchor to that of the host anchor is 1:1; if the host anchor and the guest anchor are of different genders, then a preset ratio is determined for scaling. For example, when the preset male height is 178 cm and the female height is 168 cm, it is determined to scale the guest character layer so that the ratio of the forehead width of the guest anchor to that of the host anchor is 1:0.94.

[0199] It should be understood that 1:1 and 1:0.94 can be considered as scaling parameters.

[0200] It should be noted that the numbers here are only examples and do not constitute limitations of the present application and can also be replaced by other numbers.

[0201] S809, the host host terminal continues to call the synthesis parameter control module to determine the height ratio parameter, so that the host host terminal adjusts the height position of the character in the guest character layer of the fourth screen to coordinate with the height position of the character in the host character layer of the third screen.

[0202] Exemplarily, the synthesis parameter control module determines whether the host host and the guest host are of the same gender based on face recognition technology. If they are of the same gender, the height ratio of both hosts is adjusted to 1:1; if they are of different genders, the height ratio of the exposed parts of both hosts is adjusted to the system-set height ratio. For example, when the male height is preset to be 178 cm and the female height is 168 cm, the height ratio of the exposed parts of both hosts is adjusted to 1:0.94.

[0203] The 1:1 and 1:0.94 here can be considered as height ratio parameters.

[0204] It should be noted that, like S808, the numbers here are only examples and do not constitute limitations of the present application. Other numbers may also be used.

[0205] S810, after the position parameters are determined, the layer synthesis module is called to superimpose and synthesize the background layer of the third screen, the guest character layer of the fourth screen and the host character layer of the third screen based on the zoom ratio parameter and the height ratio parameter to obtain the second target screen and output the second target screen on the host anchor end.

[0206] Specifically, in this step, the host host will superimpose the background layer, the guest character layer of the fourth screen, and the host character layer of the third screen in a bottom-up order, and adjust the fourth screen based on the zoom ratio and height ratio parameters to superimpose and synthesize the second target screen. Specifically, the obtained second target screen is the screen of the intersection area of ​​the three layers.

[0207] For example, Fig.12 A schematic diagram of scaling a guest character layer provided by an embodiment of the present application. Fig.12 As shown, before synthesis, the layer synthesis module first measures the forehead width of the host host in the host character layer and the forehead width of the guest in the guest character layer, and then uses the forehead width of the host host as a reference and completes the scaling of the guest host's character layer based on the scaling ratio parameter sent by the synthesis parameter control module, so that the ratio of the forehead width of the host host in the scaled host character layer to the forehead width of the guest host in the guest character layer is the scaling ratio. For example, when the host host and the guest host are of different genders, the synthesis parameter module determines to scale the guest character layer so that the ratio of the forehead width of the guest host to that of the host host is 1:1; when the host host and the guest host are of different genders, the synthesis parameter module determines that the preset ratio parameter is 1:0.94, for example, then for Fig.12 When the gender of the host host is male and the gender of the guest host is female, the synthesis parameter control module will determine the scaling parameter to be 1:0.94 and send the scaling parameter to the layer synthesis module. The layer synthesis module will scale the guest character layer so that the width ratio of the forehead of the host host in the host character layer and the forehead of the guest host in the scaled guest character layer is 1:0.94.

[0208] After scaling, the layer synthesis module in the host host continues to adjust the position of the guest character layer based on the position parameters, including:

[0209] Step 1: Reference Fig.13 , determine the center position of the host character layer of the host anchor, and determine the center position of the guest character layer of the guest anchor after scaling;

[0210] Step 2: Overlay the center position of the host character layer and the center position of the guest character layer, and move the guest character layer up and down based on the height ratio parameter sent by the synthesis parameter control module, so that the guest character and the host character layer are coordinated in height.

[0211] For example, the above-mentioned synthesis parameter control module determines whether the host host and the guest host are of the same gender based on face recognition technology. If they are of the same gender, the height ratio of both hosts is adjusted to 1:1; if they are of different genders, the height ratio of the exposed parts of both hosts is adjusted to the height ratio set by the system. For example, when the male height is preset to be 178cm and the female height is 168cm, the height ratio of the exposed parts of both hosts is adjusted to 1:0.94. Fig.14 , when the gender of the host anchor is male and the gender of the guest anchor is female, the synthesis parameter control module will determine the position parameter to be 1:0.94 and send the height ratio parameter to the layer synthesis module. The layer synthesis module will move the scaled guest character layer downward so that the ratio between the height of the host anchor in the host character layer and the height of the guest anchor in the scaled guest character layer is 1:0.94.

[0212] Step 4: Determine the image displayed in the intersection area of ​​the three layers as the second target image.

[0213] For example, Fig.14 The image displayed in the cross area within the black frame is the second target image.

[0214] It should be understood that by scaling and adjusting the height, the complexity of adjusting the composite image based on the person's real height, weight, and other dimensions is simplified, while meeting the host's image needs. In addition, by superimposing the three layers at the fixed position to synthesize the target image, the need for a unified background when the two hosts are connected via video is realized in a lightweight manner.

[0215] S811, the host host end pushes the second target picture to the guest host end, and the guest host end previews the second synthesized picture after receiving it.

[0216] For example, based on RTC technology, the host anchor end establishes a connection with the guest anchor end through the RTC network to push the synthesized result image to the host anchor end through the RTC protocol.

[0217] In implementation, this step can be performed by Figure 7 The host host terminal's microphone sending stream module is shown to execute the operation; correspondingly, the guest host terminal's microphone receiving stream module is executed.

[0218] Among them, S806 to S811 are referred to as the automatic negotiation stage between the host host end and the guest host end in this application. That is, the host host end automatically confirms the scaling parameters and position parameters, and then adjusts the guest character layer of the fourth screen based on the automatically confirmed scaling parameters and height ratio parameters, so that the guest character layer of the fourth screen is coordinated with the host character layer of the third screen.

[0219] In the present application, the automatic determination of the scaling ratio parameters and the determination of the position parameters by the host host end are also referred to as automatic negotiation ratio and automatic negotiation position, respectively.

[0220] In this application, when the host host terminal determines the second target screen, the second target screen will be displayed in the automatic coordination interface. Fig.10 As shown, the second target screen is also displayed on the automatic negotiation interface. It should be understood that the second target screen is the screen obtained by the host host terminal by executing the above S806 to S811. In other words, the second target screen is the screen of the overlapping area obtained by superimposing the background layer of the third screen, the character layer of the fourth screen and the character layer of the third screen in a bottom-to-top order based on the parameters set by the host host terminal. Among them, the third screen is the screen captured by the first terminal when the first user using the first terminal places the head in the designated area of ​​the left half of the screen of the first terminal, and the fourth screen is the screen captured by the second terminal when the second user using the second terminal places the head in the designated area of ​​the right half of the screen of the second terminal.

[0221] S812: If the host host and the guest host are not satisfied with the result of the automatic negotiation, a manual negotiation phase is initiated.

[0222] The host host and the guest host being dissatisfied with the result of the automatic negotiation can also be expressed as: the host host and the guest host being dissatisfied with the ratio parameter and / or position parameter determined by the host host in the automatic negotiation stage.

[0223] The manual negotiation stage refers to: the host host, for example, uses two-finger operation on the host host terminal to scale the guest character layer and manually adjust the height of the guest character layer so that both hosts are satisfied with the adjusted composite image. In other words, the automatic negotiation stage does not require user participation, while the manual negotiation stage requires user participation.

[0224] For example, when the host or guest anchor is not satisfied with the composite image during the automatic negotiation phase, the host or guest anchor clicks Fig.10 The "manual negotiation" control (also referred to as the second control) in the automatic negotiation interface shown triggers the start of manual negotiation. Fig.15 The user interface (also referred to as the manual negotiation interface in this application) displayed by the host host end after the manual negotiation is triggered by the host host end is shown. On the other hand, the guest host end also displays the manual negotiation interface. The host host can manually scale or adjust the position of the guest character layer of the guest host in the second target picture obtained in the automatic negotiation stage on the manual negotiation interface. Accordingly, the host host end displays the manually adjusted picture (also referred to as the third target picture in this application) and sends the third target picture to the guest host end.

[0225] This step is optional. Or in other words, this step may be omitted, and S813 may be directly executed after S811 is executed.

[0226] S813, after the host host end and the guest host end complete the negotiation (automatic negotiation or manual negotiation), the host host end triggers a confirmation message.

[0227] For example, if the host host and the guest host are satisfied with the second composite picture synthesized in the automatic negotiation stage, such as Fig.10 As shown, the host host and the guest host can click the confirmation control (also called the first control) in the automatic negotiation interface.

[0228] For another example, if the host anchor and the guest anchor are satisfied with the third composite picture synthesized in the manual negotiation stage, Fig.15 As shown, the host host and the guest host can click the confirmation control (also called the third control) in the manual negotiation interface.

[0229] During implementation, the host anchor can receive the host anchor's confirmation message through the microphone connection management module and send the confirmation message to the microphone connection management service module in the live broadcast service platform.

[0230] S814, the microphone connection management service module in the live broadcast service platform sends the confirmation message of the host anchor end to the guest anchor end.

[0231] S815, the guest host terminal triggers a confirmation message.

[0232] During implementation, the guest host receives the guest host's request message through the microphone connection management module and sends the confirmation message to the microphone connection management service module in the live broadcast service platform.

[0233] S816, the microphone connection management service module in the live broadcast service platform sends the confirmation message of the guest anchor to the host anchor, and the whole negotiation process is completed.

[0234] S817, the host host end calls the live streaming push module, pushes the superimposed and synthesized picture to the live media module of the live broadcast service platform to start the live broadcast for the audience watching the host host. On the other hand, the guest host end also calls the live streaming push module, collects the synthesized picture sent by the guest host end's microphone receiving stream module, and pushes the superimposed and synthesized picture to the live media module of the live broadcast service platform to start the live broadcast for the audience watching the guest host.

[0235] In this application, after the entire negotiation process is completed, during the live broadcast process, the zoom ratio parameters and / or height ratio parameters determined during the negotiation process are used, that is, during the live broadcast process, when the host background layer of the host host end screen, the host character layer of the host host end screen, and the guest character layer of the guest host end screen are superimposed and synthesized, the zoom ratio parameters and / or position parameters determined in the negotiation stage are used. On the other hand, after the entire negotiation process is completed, the host host and the guest host can move freely.

[0236] same Figure 6 The same as the above embodiment, during the live broadcast of the host host and the guest host, the synthesis parameter control module of the host host can regularly collect the host character layer and the guest character layer to determine whether the guest character layer is coordinated with the host character layer (that is, determine whether the host host and the guest host are coordinated) to adjust the occlusion relationship.

[0237] For example, when both hosts confirm the synthesis parameters and start to synthesize and push the stream, the default occlusion relationship of the synthesis parameter control module is that the host host occludes the guest live broadcast (i.e., the layers from bottom to top are the background layer of the host host's screen> the guest character layer> the host character layer). Afterwards: the synthesis parameter control module collects the forehead widths of the host and guest character layers from the matting module periodically (such as a 1s period), calculates the ratio of the forehead widths of the host host and the guest host (host forehead width: guest forehead width) and the first threshold value, and determines the occlusion relationship based on the size relationship.

[0238] For example: taking the example that the host anchor and the guest anchor have the same gender: when the forehead width ratio is greater than or equal to 1:1*(1+10%), the synthesis parameter control module sends the occlusion relationship that the host anchor occludes the guest anchor to the layer synthesis module; when the forehead width ratio is less than 1:1*(1+10%), and the host anchor end is set to allow the guest anchor to occlude the host anchor, the synthesis parameter control sends the occlusion relationship that the guest anchor occludes the host anchor to the layer synthesis module; when the forehead width ratio is less than 1:1*(1+10%), and the host anchor end is set to not allow the guest anchor to occlude the host anchor, the synthesis parameter control sends the cancellation of the guest character layer synthesis strategy to the layer synthesis module.

[0239] Optionally, on the basis of sending the cancellation strategy of guest character layer synthesis to the layer synthesis module, the synthesis parameter control module notifies the microphone connection management module that the guest anchor is too close to the camera, and the host anchor microphone connection management module sends the notification that the guest anchor is too close to the camera to the live broadcast platform-microphone connection management service module. The live broadcast platform-microphone connection management service module notifies the microphone connection management module of the guest anchor end that the guest anchor is too close to the camera, and the microphone connection management module of the guest anchor end prompts the guest to adjust his position in time.

[0240] S818, the audience of the host anchor and the guest anchor enter their respective live broadcast rooms, and pull the corresponding live broadcast screen (also called live broadcast stream) from the live broadcast service platform for viewing.

[0241] In the video linking method provided in this embodiment, the host anchor end will first determine the overlay parameters (including zoom ratio parameters and / or height ratio parameters) used for overlaying and synthesizing the background layer of the picture collected by the host anchor end, the character layer of the picture collected by the host anchor end, and the character layer of the picture collected by the guest anchor end through automatic negotiation or manual negotiation. Then, during the live broadcast process, the overlay parameters are fixed, and the occlusion relationship between the character layer of the picture collected by the host anchor end and the character layer of the picture collected by the guest anchor end when synthesizing the picture is determined. Then, based on the overlay parameters and the occlusion relationship, the picture of the live broadcast room when the host anchor and the guest anchor are video linked is synthesized and displayed to the audience end.

[0242] It should be understood that the video linking method provided in this application is applicable to live broadcast scenarios with fixed-position camera and fixed focal length. For example, it can be a live broadcast method in which the host uses a PC host terminal + fixed camera to collect. For another example, it can be a live broadcast method in which the host uses a mobile phone host terminal with a fixed position. Both methods can provide live broadcast collection capabilities with fixed-position camera and fixed focal length.

[0243] Optionally, the video linking method provided by the present application may also be executed by a live broadcast service platform (also referred to as a live broadcast server). For example, when the live broadcast platform server executes the video linking method, it includes:

[0244] 1) The live broadcast service platform obtains a first picture and a second picture, where the first picture includes a picture captured by a first terminal, and the second picture includes a picture captured by a second terminal, where the second terminal is a terminal connected to the first terminal via video.

[0245] Similar to step S501, in this embodiment, the picture captured by the camera of the first terminal when the first user and the second user are connected via video is called the first picture (that is, the image picture of the physical space where the first user is located captured by the camera of the first terminal), and the picture captured by the camera of the second terminal when the second user and the first user are connected via video is called the second picture (that is, the image picture of the physical space where the second user is located captured by the camera of the second terminal).

[0246] During implementation, the first terminal sends the first picture to the live broadcast service platform, and the second terminal sends the captured second picture to the live broadcast service platform.

[0247] 2) The live broadcast service platform extracts the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture, and superimposes the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture to obtain a first target picture, wherein, when superimposing, the background layer of the first picture is located below the character layer of the first picture and the character layer of the second picture.

[0248] Among them, this embodiment and Figure 5 The difference between the embodiments shown is that the operations of obtaining the first picture and the second picture, extracting the background layer of the first picture, the character layer of the first picture and the character layer of the second picture, and superimposing the background layer of the first picture, the character layer of the first picture and the character layer of the second picture to obtain the first target picture are performed by the live broadcast service platform. For the description of the rest, please refer to Figure 5 The embodiments shown will not be described in detail here.

[0249] In some embodiments, the first target picture obtained by the live broadcast service platform is a picture of an overlapping area after superimposing the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture.

[0250] In some embodiments, the live broadcast service platform may also first determine the overlay order of the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture; and then overlay the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture based on the overlay order to obtain the first target picture.

[0251] In some embodiments, the live broadcast service platform determines the order of superimposing the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen: obtain the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen; if the ratio between the forehead width of the first user and the forehead width of the second user is greater than or equal to the first threshold, determine to superimpose the background layer of the first screen, the character layer of the second screen, and the character layer of the first screen in the order from bottom to top.

[0252] In some embodiments, the live broadcast service platform determines the order of superimposing the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen, including: obtaining the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen; if the ratio between the forehead width of the first user and the forehead width of the second user is less than a first threshold value, and the character layer of the second screen is allowed to be above the character layer of the first screen, determine to superimpose the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen in the order from bottom to top.

[0253] In some embodiments, the live broadcast service platform determines the order of superimposing the background layer of the first screen, the character layer of the first screen, and the character layer of the second screen, including: obtaining the forehead width of the first user in the character layer of the first screen and the forehead width of the second user in the character layer of the second screen; if the ratio of the forehead width of the first user to the forehead width of the second user is less than a first threshold, and the character layer of the second screen is not allowed to be above the character layer of the first screen, superimposing the background layer of the first screen and the character layer of the first screen in the order from bottom to top.

[0254] In some embodiments, when the live broadcast platform server superimposes the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture to obtain the first target picture, it can also superimpose the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture based on the superposition parameters to obtain the first target picture, and the superposition parameters include at least one of the following parameters: a parameter indicating the scaling ratio between the character layer of the second picture and the character layer of the first picture, and a parameter indicating the height ratio between the character layer of the second picture and the character height in the character layer of the first picture.

[0255] For the description of the superposition parameters, reference may be made to the description in the aforementioned embodiment and will not be repeated here.

[0256] In one implementation, the superposition parameters are set in advance, for example, in advance in the host host terminal, or in the live broadcast service platform. When set in the host host terminal, the host host terminal can send the superposition parameters so that the live broadcast service platform knows the superposition parameters.

[0257] For example, when the guest host agrees to connect with the host host, the host host and the guest host will display Fig.10 In the automatic negotiation interface shown, the host anchor end sends the picture (i.e., the third picture) collected when the host anchor's head is in the specified area to the anchor platform server, and the guest anchor end sends the picture (i.e., the fourth picture) collected when the guest anchor's head is in the specified area to the anchor platform server, and then the anchor server platform overlays the background layer of the third picture, the character layer of the fourth picture, and the character layer of the third picture in a bottom-up order based on the above-mentioned pre-set overlay parameters to obtain the picture of the overlapping area (i.e., the second target picture) and sends the second target picture to the host anchor end and the guest anchor end respectively; thereafter, if the host anchor and the guest anchor are satisfied with the second target picture and trigger the start of the live broadcast to the audience end, for example, the host anchor and the guest anchor both click Fig.10After the confirmation control in the automatic negotiation interface shown triggers the start of live broadcast to the audience, in the subsequent video live broadcast process, the live broadcast service platform performs overlay synthesis based on the pre-set overlay parameters, and sends the synthesized images to the host end performing the video live broadcast.

[0258] In another implementation, the above-mentioned superposition parameters are determined based on the operations performed by the host anchor on the host anchor terminal.

[0259] For example, when the host host and the guest host are not satisfied with the second target image synthesized by the live broadcast service platform based on the pre-set superposition parameters, at this time, the host host triggers the manual negotiation stage through the host host and / or the guest host triggers the manual negotiation stage through the guest host. For example, the host host and / or the guest host can click Fig.10 The manual negotiation control in the automatic negotiation interface shown is used to trigger the manual negotiation stage. Accordingly, in response to the host host and / or the guest host clicking Fig.10 The manual negotiation controls in the automatic negotiation interface shown in the figure are displayed on the host host side and the guest host side. Fig.15 The manual negotiation interface is shown, and then the host anchor manually adjusts the guest character layer in the displayed second target screen to satisfy the host anchor and the guest anchor; after that, in the subsequent video live broadcast process, the live broadcast service platform performs overlay synthesis based on the manually adjusted overlay parameters that satisfy the connected anchors, and sends the synthesized pictures to the anchor ends performing the video live broadcast.

[0260] Fig.16 A schematic diagram of the structure of a video linking device of a terminal device provided in one embodiment of the present application. Fig.16 As shown, the video link device 1600 includes an acquisition module 1601 and a processing module 1602 .

[0261] As an example, the acquisition module 1601 can be used to execute Figure 5 In the embodiment shown, the operation of obtaining the first picture and the second picture, or for executing Figure 8 Operations of the third and fourth screens in the illustrated embodiment.

[0262] As another example, the processing module 1602 may be configured to perform operations of extracting the first picture, the character layer of the first picture, and the character layer of the second picture.

[0263] As another example, the processing module 1602 may be used to perform an operation of superimposing a background layer of a first picture, a character layer of the first picture, and a character layer of a second picture to obtain a first target picture.

[0264] Fig.17This is a schematic diagram of the structure of a video linking device provided by another embodiment of the present application. Fig.17 As shown, the processing device 1700 may include a processor 1701 and an interface circuit 1702. The processor 1701 and the interface circuit 1702 are coupled to each other. It is understood that the interface circuit 1702 may be a transceiver or an input-output interface. Optionally, the processing device 1700 may also include a memory 1703 for storing instructions executed by the processor 1701 or storing input data required by the processor 1701 to execute instructions or storing data generated after the processor 1701 executes instructions.

[0265] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0266] The method steps in the embodiments of the present application can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also be present in a network device or a terminal device as discrete components.

[0267] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc; it may also be a semiconductor medium, for example, a solid-state hard disk.

[0268] In the various embodiments of the present application, unless otherwise specified or provided in a logical conflict, the terms and / or descriptions between the different embodiments are consistent and may be referenced to each other, and the technical features in the different embodiments may be combined to form new embodiments according to their inherent logical relationships.

[0269] It is understood that the various numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

Claims

1. A video linking method, characterized in that: include: Acquire a first picture, where the first picture includes a picture captured by the first terminal; Acquire a second picture, where the second picture comes from a second terminal, and the second terminal is a terminal that is connected to the first terminal via video link; Extracting the character layer of the second picture; The background layer of the first picture, the character layer of the first picture and the character layer of the second picture are superimposed to obtain a first target picture, wherein the character layer of the first picture and the character layer of the second picture are both superimposed under the background layer of the first picture.

2. The method according to claim 1, characterized in that The first target picture is a picture of an overlapping area after the background layer of the first picture, the character layer of the first picture, and the character layer of the second picture are superimposed.

3. The method according to claim 1 or 2, characterized in that: The superimposing the background layer of the first picture, the character layer of the first picture and the character layer of the second picture comprises: Acquire a forehead width of a first user in the character layer of the first screen and a forehead width of a second user in the character layer of the second screen; If the ratio of the forehead width of the first user to the forehead width of the second user is greater than or equal to a first threshold, the background layer of the first screen, the character layer of the second screen, and the character layer of the first screen are superimposed in order from bottom to top.

4. The method according to claim 1 or 2, characterized in that: The superimposing the background layer of the first picture, the character layer of the first picture and the character layer of the second picture comprises: Acquire a forehead width of a first user in the character layer of the first screen and a forehead width of a second user in the character layer of the second screen; If the ratio of the forehead width of the first user to the forehead width of the second user is less than a first threshold, and the character layer of the second screen is allowed to be above the character layer of the first screen, the background layer of the first screen, the character layer of the first screen and the character layer of the second screen are superimposed in order from bottom to top.

5. The method according to claim 1 or 2, characterized in that: The superimposing the background layer of the first picture, the character layer of the first picture and the character layer of the second picture comprises: Acquire a forehead width of a first user in the character layer of the first screen and a forehead width of a second user in the character layer of the second screen; If the ratio of the width of the forehead of the first user to the width of the forehead of the second user is less than a first threshold, and the character layer of the second screen is not allowed to be above the character layer of the first screen, the background layer of the first screen and the character layer of the first screen are superimposed in order from bottom to top.

6. The method according to any one of claims 1 to 5, characterized in that When the background layer of the first picture, the character layer of the first picture and the character layer of the second picture are superimposed to obtain a first target picture, the method further includes: The background layer of the first picture, the character layer of the first picture and the character layer of the second picture are superimposed based on superposition parameters to obtain the first target picture, and the superposition parameters include at least one of the following parameters: a parameter indicating the scaling ratio between the character layer of the second picture and the character layer of the first picture, and a parameter indicating the height ratio between the character layer of the second picture and the character height in the character layer of the first picture.

7. The method according to claim 6, characterized in that Applied to the first terminal, the method further includes: Sending a microphone connection request message, where the microphone connection request message is used to request a video microphone connection with the second terminal; In response to a confirmation message that the second terminal agrees to conduct a video link, an automatic negotiation interface is output, wherein the automatic negotiation interface includes a second target screen and a first control, wherein the second target screen is a screen of an overlapping area obtained by superimposing a background layer of a third screen, a character layer of a fourth screen, and a character layer of the third screen in a bottom-up order by the first terminal, and characters in the character layer of the third screen included in the second target screen partially overlap with characters in the character layer of the fourth screen; In response to an operation of clicking the first control, determining a second parameter as the superposition parameter; The second target picture can be obtained by using the second parameter to superimpose the background layer of the third picture, the character layer of the fourth picture and the character layer of the third picture in order from bottom to top.

8. The method according to claim 7, characterized in that The automatic negotiation interface also includes a second control, and the method further includes: In response to the operation of clicking the second control, a manual negotiation interface is displayed, wherein the manual negotiation interface includes the second target screen; In response to adjusting the character layer of the fourth picture in the second target picture on the manual negotiation interface, displaying the adjusted third target picture; When the first user and the second user are satisfied with the third target picture, determining the first parameter as the superposition parameter; The third target picture can be obtained when the character layer of the fourth picture in the second target picture is adjusted using the first parameter.

9. The method according to claim 8, characterized in that The manual negotiation interface includes a third control, and the second terminal and the first terminal display the same manual negotiation interface; The determining that the first user and the second user are satisfied with the third target picture includes: If the first user clicks the third control on the first terminal and the second user clicks the third control in the manual negotiation interface on the second terminal, it is determined that the first user and the second user are satisfied with the third target screen.

10. The method according to any one of claims 1 to 9, characterized in that When the background layer of the first picture, the character layer of the first picture and the character layer of the second picture are superimposed to obtain a first target picture, the method includes: Scaling the character layer of the second picture based on a parameter indicating a scaling ratio between the character layer of the second picture and the character layer of the first picture; Aligning the center point of the character layer in the first picture with the center point of the character layer in the scaled second picture; Based on a parameter indicating a height ratio between the character layer of the second picture and the character heights in the character layer of the first picture, the scaled character layer of the second picture is moved up and down so that the height ratio between the character layer of the second picture and the character heights in the character layer of the first picture is satisfied.

11. A video link device, characterized in that: The method comprises a functional module for implementing the method according to any one of claims 1 to 10.

12. A video connection device, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to execute program instructions in the memory to implement the method according to any one of claims 1 to 10.

13. An electronic device, characterized in that: Including the video connection device as described in claim 11 or claim 12.

14. A live broadcast server, characterized in that: The method comprises a functional module for implementing the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for computer execution, wherein the program code includes instructions for implementing the method according to any one of claims 1 to 10.

16. A computer program product, characterized in that The computer program product comprises instructions for implementing the method according to any one of claims 1 to 10.

Images