Hybrid live streaming methods, devices, equipment and storage media

Abstract

The present invention relates to the field of live broadcast technology, and discloses a hybrid live broadcast method, device, equipment and storage medium. The method comprises: obtaining the real-time live broadcast content of the live broadcast space to be broadcasted, and respectively capturing the original scene elements in each real-time live broadcast content; according to the live broadcast scene parameters corresponding to the live broadcast space, the original scene elements are coordinated to adjust the spatial effects to obtain the final scene elements; the final scene elements are synthesized with the real-time live broadcast content of the target live broadcast space to obtain hybrid live broadcast content, and the hybrid live broadcast content is broadcasted in the target live broadcast space. The present invention realizes the simultaneous display of the contents of two physical spaces on the same live broadcast page, and realizes the hybrid live broadcast of elements of real and virtual spaces.

Description

Technical Field

[0001] The present invention relates to the field of live broadcast technology, and in particular to a hybrid live broadcast method, device, equipment and storage medium. Background Art

[0002] With the rapid development of network technology in recent years, online live broadcast has gradually replaced TV live broadcast and become the mainstream live broadcast method. The concept of live broadcast has also been expanded, and the methods and styles of live broadcast have emerged in an endless stream. With the support of underlying technology, the anchor can apply more interactive functions during the live broadcast process, providing the audience with more exciting and diverse live content, giving users a better live broadcast experience.

[0003] When live broadcasting involves mixed live broadcasting of scenes from different live broadcasting spaces, the current live broadcasting scenes usually adopt the following methods: recording and displaying based on the scene range obtained by the user during the actual live broadcast, or manually recording the screen by the user, and then splicing the real and virtual content in the same live broadcast room. However, it is impossible to display the content of two physical spaces on one live broadcast page at the same time during the live broadcast. Summary of the invention

[0004] The main purpose of the present invention is to solve the problem that it is impossible to display the contents of two physical spaces simultaneously on a live broadcast page during a live broadcast process.

[0005] A first aspect of the present invention provides a hybrid live broadcast method, comprising: obtaining real-time live broadcast content of a live broadcast space to be broadcast, and capturing original scene elements in the real-time live broadcast content; according to the live broadcast scene parameters corresponding to the live broadcast space, collaboratively adjusting the spatial effects of the original scene elements to obtain final scene elements; synthesizing the final scene elements with the real-time live broadcast content of the target live broadcast space to obtain hybrid live broadcast content, and rebroadcasting the hybrid live broadcast content in the target live broadcast space.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the coordinated adjustment of the spatial effects of the original scene elements according to the live scene parameters corresponding to the live space to obtain the final scene elements includes: extracting first camera parameters and first scene rendering information from the live scene parameters corresponding to the live space; calculating the actual size of the original scene elements according to the first camera parameters, scaling the original scene elements to the actual size, and performing rendering effect culling on the original scene elements according to the first scene rendering information to obtain the final scene elements.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the synthesizing the final scene element with the real-time live content of the target live space to obtain mixed live content includes: determining the second scene rendering information corresponding to the target live space, and the first space transformation parameters between the live space; according to the first space transformation parameters, adjusting the position of the final scene element in the target live space, and adapting and placing it according to the corresponding live terminal type, and mapping the second scene rendering information to the adapted and placed scene element; obtaining each video frame corresponding to the real-time live content of the target live space, and time-aligning the mapped scene element with each video frame; according to the result of the time alignment, synthesizing the final scene element with each video frame to obtain a new video frame, and generating mixed live content based on the new video frame.

[0008] Optionally, in a third implementation method of the first aspect of the present invention, the position of the final scene element in the target live broadcast space is adjusted according to the first spatial transformation parameter, and the scene element is adaptively placed according to the corresponding live broadcast terminal type. The steps include: adjusting the position of the final scene element in the target live broadcast space according to the first spatial transformation parameter, and identifying the first live broadcast terminal type corresponding to the target live broadcast space and the second live broadcast terminal type corresponding to the live broadcast space; obtaining a first interface size corresponding to the first live broadcast terminal type and a second interface size corresponding to the second live broadcast terminal type; calculating an interface mapping relationship between the first interface size and the second interface size, and placing the scene elements after the position adjustment in the target live broadcast space according to the interface mapping relationship.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the coordinated adjustment of the spatial effects of the original scene elements according to the live broadcast scene parameters corresponding to the live broadcast space to obtain the final scene elements includes: when the live broadcast space includes a target live broadcast space and other live broadcast spaces other than the target live broadcast space, extracting second camera parameters and third scene rendering information from the live broadcast scene parameters corresponding to the target live broadcast space, and extracting third camera parameters and fourth scene rendering information from the live broadcast scene parameters corresponding to the other live broadcast spaces; constructing a mapping relationship between the target live broadcast space and the other live broadcast spaces according to the second camera parameters and the third camera parameters, and scaling the original scene elements according to the mapping relationship; determining the second spatial transformation parameters between the target live broadcast space and the other live broadcast spaces, and adjusting the position of the scaled original scene elements in the target live broadcast space according to the second spatial transformation parameters, and adapting and placing them according to the corresponding live broadcast terminal type; performing rendering effect culling on the adapted and placed original scene elements according to the fourth scene rendering information, and mapping the third scene rendering information to the original scene elements after the rendering effect is culled, to obtain the final scene elements.

[0010] Optionally, in a fifth implementation method of the first aspect of the present invention, synthesizing the final scene element with the real-time live content of the target live space to obtain mixed live content includes: determining each video frame corresponding to the real-time live content of the target live space, and time-aligning the final scene element with the each video frame; according to the result of the time alignment, synthesizing the final scene element with the each video frame to obtain a new video frame, and generating mixed live content based on the new video frame.

[0011] Optionally, in a sixth implementation method of the first aspect of the present invention, before the collaborative adjustment of the spatial effects of the original scene elements according to the live scene parameters corresponding to the live space to obtain the final scene elements, it also includes: in response to the dragging instructions of the original scene elements on the live terminal interface, determining the dragging trajectory of the original scene elements, and generating corresponding spatial transformation parameters based on the dragging trajectory; and / or identifying the scene matching information of the original scene elements on the live terminal interface, and the initial position information of the original scene elements and the scene matching information, generating corresponding spatial transformation parameters; the spatial transformation parameters include the first spatial transformation parameters and the second spatial transformation parameters.

[0012] Optionally, in a seventh implementation method of the first aspect of the present invention, capturing the original scene elements in the real-time live broadcast content includes: respectively identifying the interface elements in each of the real-time live broadcast contents, and intercepting each identified interface element; in response to a selection instruction of a selection button corresponding to each of the interface elements on the live broadcast terminal interface, capturing the target interface element and serving it as the original scene element in each of the real-time live broadcast contents.

[0013] Optionally, in an eighth implementation manner of the first aspect of the present invention, capturing the original scene elements in the real-time live broadcast content includes: determining a cropping range for each of the real-time live broadcast contents in response to cropping instructions for each of the real-time live broadcast contents on the live broadcast terminal interface; and using local real-time live broadcast content within the cropping range as the original scene elements in each of the real-time live broadcast contents.

[0014] The second aspect of the present invention provides a hybrid live broadcast device, including: a capture module, used to obtain real-time live broadcast content of a live broadcast space to be broadcast, and capture original scene elements in the real-time live broadcast content; an adjustment module, used to coordinate the spatial effects of the original scene elements according to the live broadcast scene parameters corresponding to the live broadcast space, to obtain final scene elements; a synthesis module, used to synthesize the final scene elements with the real-time live broadcast content of the target live broadcast space, to obtain hybrid live broadcast content, and to broadcast the hybrid live broadcast content in the target live broadcast space.

[0015] Optionally, in a first implementation method of the second aspect of the present invention, the adjustment module includes a first processing unit, which is used to: extract first camera parameters and first scene rendering information from the live scene parameters corresponding to the live space; calculate the actual size of the original scene element according to the first camera parameters, scale the original scene element to the actual size, and perform rendering effect culling on the original scene element according to the first scene rendering information to obtain the final scene element.

[0016] Optionally, in a second implementation method of the second aspect of the present invention, the synthesis module includes a first synthesis unit, which is used to: determine second scene rendering information corresponding to the target live broadcast space, and a first space transformation parameter between the live broadcast space; according to the first space transformation parameter, adjust the position of the final scene element in the target live broadcast space, and adapt and place it according to the corresponding live broadcast terminal type, and map the second scene rendering information to the adapted and placed scene element; obtain each video frame corresponding to the real-time live broadcast content of the target live broadcast space, and time-align the mapped scene elements with the each video frame; according to the result of the time alignment, synthesize the final scene element with the each video frame to obtain a new video frame, and generate mixed live broadcast content based on the new video frame.

[0017] Optionally, in a third implementation of the second aspect of the present invention, the first synthesis unit is further used for:

[0018] According to the first space transformation parameter, the position of the final scene element is adjusted in the target live broadcast space, and a first live broadcast terminal type corresponding to the target live broadcast space and a second live broadcast terminal type corresponding to the live broadcast space are identified;

[0019] Obtaining a first interface size corresponding to the first live broadcast terminal type and a second interface size corresponding to the second live broadcast terminal type;

[0020] An interface mapping relationship between the first interface size and the second interface size is calculated, and the scene elements after position adjustment are placed in the target live broadcast space according to the interface mapping relationship.

[0021] Optionally, in a fourth implementation method of the second aspect of the present invention, the adjustment module also includes a second processing unit, which is used to: when the live broadcast space includes a target live broadcast space and other live broadcast spaces other than the target live broadcast space, extract second camera parameters and third scene rendering information from the live broadcast scene parameters corresponding to the target live broadcast space, and extract third camera parameters and fourth scene rendering information from the live broadcast scene parameters corresponding to the other live broadcast spaces; construct a mapping relationship between the target live broadcast space and the other live broadcast spaces according to the second camera parameters and the third camera parameters, and scale the original scene elements according to the mapping relationship; determine the second space transformation parameters between the target live broadcast space and the other live broadcast spaces, and according to the second space transformation parameters, adjust the position of the scaled original scene elements in the target live broadcast space, and adapt and place them according to the corresponding live broadcast terminal type; perform rendering effect culling on the adapted and placed original scene elements according to the fourth scene rendering information, and map the third scene rendering information to the original scene elements after the rendering effect is culled, to obtain the final scene elements.

[0022] Optionally, in a fifth implementation of the second aspect of the present invention, the synthesis module also includes a second synthesis unit, which is used to: determine each video frame corresponding to the real-time live content of the target live space, and time-align the final scene elements with the each video frame; according to the result of the time alignment, synthesize the final scene elements with the each video frame to obtain a new video frame, and generate mixed live content based on the new video frame.

[0023] Optionally, in a sixth implementation manner of the second aspect of the present invention, the hybrid live broadcast device also includes a trajectory recognition module, which is used to: determine the dragging trajectory of the original scene element in response to the dragging instructions of the original scene element on the live broadcast terminal interface, and generate corresponding spatial transformation parameters based on the dragging trajectory; and / or identify the scene matching information of the original scene element on the live broadcast terminal interface, and the initial position information of the original scene element and the scene matching information, and generate corresponding spatial transformation parameters; the spatial transformation parameters include the first spatial transformation parameters and the second spatial transformation parameters.

[0024] Optionally, in a seventh implementation method of the second aspect of the present invention, the capture module includes an identification and capture unit, which is used to: respectively identify the interface elements in each of the real-time live broadcast contents, and intercept each identified interface element; in response to a selection instruction of a selection button corresponding to each of the interface elements on the live broadcast terminal interface, capture the target interface element and use it as the original scene element in each of the real-time live broadcast contents.

[0025] Optionally, in an eighth implementation manner of the second aspect of the present invention, the capture module also includes a cropping capture unit, which is used to: determine the cropping range of each of the real-time live content in response to the cropping instructions for each of the real-time live content on the live terminal interface; and use the local real-time live content within the cropping range as the original scene element in each of the real-time live content.

[0026] The third aspect of the present invention provides a hybrid live broadcast device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the hybrid live broadcast device executes the above-mentioned hybrid live broadcast method.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the above-mentioned hybrid live broadcast method.

[0028] In the technical solution provided by the present invention, the original scene elements are captured by capturing the real-time live content in different live spaces, such as the shooting content of multiple live rooms or multiple cameras; each original scene element contains the environment rendering data under the corresponding live space, or the actual size scaling degree caused by different focal lengths, so each original scene element is uniformly and collaboratively adjusted for spatial effects to obtain standard scene elements, which are synthesized with the real-time live content of the target live space to be broadcast, and the final scene elements are integrated into the real-time live content, and then rebroadcast in the target live space. The same live page can display the contents of two physical spaces at the same time, and the elements of real and virtual spaces can realize mixed live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of a first embodiment of a hybrid live broadcast method in an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of a second embodiment of the hybrid live broadcast method in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of an embodiment of mixed live broadcasting in different live broadcasting spaces according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of an embodiment of a hybrid live broadcast device in an embodiment of the present invention;

[0033] Figure 5 It is a schematic diagram of another embodiment of the hybrid live broadcast device in the embodiment of the present invention;

[0034] Figure 6It is a schematic diagram of an embodiment of a hybrid live broadcast device in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention provide a hybrid live broadcast method, device, equipment and storage medium, which obtain the real-time live broadcast content of the live broadcast space to be broadcasted, and respectively capture the original scene elements in each real-time live broadcast content; according to the live broadcast scene parameters corresponding to the live broadcast space, the original scene elements are coordinated to adjust the spatial effects to obtain the final scene elements; the final scene elements are synthesized with the real-time live broadcast content of the target live broadcast space to obtain hybrid live broadcast content, and the hybrid live broadcast content is rebroadcasted in the target live broadcast space. The present invention realizes that the same live broadcast page displays the contents of two physical spaces at the same time, and realizes the hybrid live broadcast of elements of real and virtual spaces.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the hybrid live broadcast method in the embodiment of the present invention includes:

[0038] 101. Acquire real-time live broadcast content of a live broadcast space to be broadcast, and capture original scene elements in the real-time live broadcast content;

[0039] It is understandable that the execution subject of the present invention may be a hybrid live broadcast device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0040] In this embodiment, multiple live broadcast spaces obtain real-time live broadcast content through camera equipment, but are not broadcast on the live broadcast terminal. Here, such real-time live broadcast content is intercepted, and the elements of the real-time live broadcast content of different live broadcast spaces are mixed and then broadcast to the live broadcast terminal required for broadcast. Then, for each original scene element in the real-time live broadcast content, it is possible to intelligently identify the scene and capture the corresponding original scene element, or it is possible to identify each element and then provide a selection button for the user to check / click to select.

[0041] The live broadcast space can be the actual live broadcast space in the anchor's live broadcast room, or the actual live broadcast space captured by different camera equipment in the anchor's live broadcast room. For the former, for example, a selection button for other anchor's live broadcast rooms is provided in each live broadcast terminal interface to establish a connection with them and execute the mixed live broadcast function to obtain their real-time live broadcast content. For the latter, for example, a live broadcast room sets up camera equipment to shoot the live broadcast space in different scenes such as indoor and outdoor scenes, and selects the required live broadcast space to obtain its real-time live broadcast content.

[0042] 102. According to the live broadcast scene parameters corresponding to the live broadcast space, coordinately adjust the space effect of the original scene elements to obtain final scene elements;

[0043] In this embodiment, since different live broadcast spaces are located in different live broadcast scenes, the rendering data corresponding to each original scene element is also different. When the original scene elements of different live broadcast scenes are integrated, there may be incoordination problems. Therefore, it is necessary to unify the spatial effects of the original scene elements in different scenes according to the parameters of each live broadcast scene, and coordinate them to the same live broadcast scene as standard available scene elements.

[0044] Specifically, the live broadcast scene parameters include at least the focal length parameters of the live broadcast terminal in the live broadcast space, the rendering data of the original scene elements, and may also include the spatial position information and viewing angle of the scene, and coordinately adjust the spatial effects of the corresponding items according to the live broadcast scene parameters.

[0045] Among them, for the focal length parameter, the size effect (i.e., dimension) of the original scene element can be adjusted; for the rendering data, the light, saturation, color, etc. of the original scene element can be removed or adjusted; for the spatial position information, the placement of the original scene element, the terminal type adaptation mapping, etc. can be adjusted; for the viewing angle, the spatial display angle of the original scene element can be adjusted, such as three-dimensional arbitrary angle rotation. In this way, the adjustment of the original scene element is completed, and the final scene element is obtained, which can be used to integrate the live broadcast content for mixed live broadcast without causing inharmonious spatial effects.

[0046] 103. Synthesize the final scene element with the real-time live content of the target live space to obtain mixed live content, and broadcast the mixed live content in the target live space.

[0047] In this embodiment, the real-time live content of the target live space is selected, and the original scene elements are extracted from multiple live spaces and processed in coordination to obtain the corresponding final scene elements. Here, it is necessary to determine the recording time of each frame in the real-time live content and the recording time of the scene element, and the two need to be matched, and then the live content and scene elements obtained at the same time are synthesized into the same frame, and then the synthesized new live content is generated into mixed live content, so that the mixed live content can achieve synchronous broadcasting of scene elements in different live spaces.

[0048] In addition, the live broadcast target space can be selected from the aforementioned live broadcast space, or it can be other live broadcast spaces except the aforementioned live broadcast space. For the two different target live broadcast spaces, the coordinated adjustment of the spatial effects of the previous original scene elements and the synthesis of scene elements and real-time live broadcast content here are different in the processing flow. Since the steps of adjusting the placement of scene elements, terminal type adaptation mapping, spatial display angle, light, saturation, color, etc. need to be implemented by using the live broadcast scene parameters of the target live broadcast space when the target live broadcast space is determined, if the target live broadcast space is selected from the aforementioned live broadcast space, it can be executed together when the spatial effects of the original scene elements are coordinated and adjusted; if the target live broadcast space is in the aforementioned live broadcast space, it is necessary to determine the target live broadcast space during the synthesis process before executing it.

[0049] In the embodiment of the present invention, the original scene elements are captured by capturing the real-time live content in different live spaces, such as the shooting content of multiple live rooms or multiple cameras; each original scene element contains the environment rendering data under the corresponding live space, or the actual size scaling degree caused by different focal lengths, so each original scene element is uniformly and collaboratively adjusted for spatial effects to obtain standard scene elements, which are synthesized with the real-time live content of the target live space to be broadcast, and the final scene elements are integrated into the real-time live content, and then rebroadcast in the target live space. This allows the same live page to display the contents of two physical spaces at the same time, and the elements of the real and virtual spaces can achieve mixed live broadcasting.

[0050] See also Figure 2 The second embodiment of the hybrid live broadcast method in the embodiment of the present invention includes:

[0051] 201. Acquire real-time live broadcast content of a live broadcast space to be broadcast, and capture original scene elements in the real-time live broadcast content;

[0052] In this embodiment, the original scene elements in the real-time live content can be captured by custom means, including: providing a selection button for the user to select the required original scene elements on the front end for capture, or providing a cropping tool to capture the original scene elements, or the previous real-time live content only shoots one original scene element, and here the entire real-time live content is directly captured to obtain the corresponding original scene element. For the three different capture methods, the first capture method is more convenient, the second capture method is more flexible, and the third capture method is simpler, and the capture methods can be arranged according to different needs.

[0053] Specifically, the first method of grabbing original scene elements is provided below, as shown below:

[0054] 1.11) Respectively identifying the interface elements in each of the real-time live broadcast contents, and intercepting each identified interface element;

[0055] 1.12) In response to a selection instruction of a selection button corresponding to each of the interface elements on the live broadcast terminal interface, a target interface element is captured and used as an original scene element in each of the real-time live broadcast contents.

[0056] In this embodiment, for each frame of interface image in the real-time live broadcast content, all interface elements contained in the live broadcast interface image are intelligently identified, such as tables, chairs, tea cups, etc., and corresponding selection buttons are generated for each interface element on the front end and displayed at a preset position; the user can capture the selected interface element by checking the selection button corresponding to the required interface element in the live broadcast interface and then submitting it to the background to obtain the original scene element.

[0057] Specifically, the second method of grabbing original scene elements is provided below, as shown below:

[0058] 1.21) In response to the cropping instructions for each of the real-time live content on the live terminal interface, determine the cropping range for each of the real-time live content;

[0059] 1.22) Using the local real-time live content within the cropping range as the original scene element in each of the real-time live content.

[0060] In this embodiment, a cropping tool is provided, and the user can select the interface element by outlining the required part of the live broadcast content on the live broadcast interface, or each live broadcast content of the angle and perspective selected previously is an interface element, and then the range of the boundary element is accurately outlined to obtain the original scene element.

[0061] 202. In response to the dragging instructions for the original scene elements on the live terminal interface, determine the dragging trajectory of the original scene elements, and generate corresponding space transformation parameters according to the dragging trajectory; and / or,

[0062] 203. Identify scene matching information of the original scene element on the live terminal interface, and generate corresponding space transformation parameters based on the initial position information of the original scene element and the scene matching information; the space transformation parameters include the first space transformation parameters and the second space transformation parameters;

[0063] In this embodiment, the live scene parameters at least include space transformation parameters, wherein the space transformation parameters can be generated by using one of the adjustment methods of dragging trajectory and scene position matching or a combination of the two adjustment methods. The first adjustment method generates space transformation parameters including the dragging trajectory information of the original scene element, and the second adjustment method generates space transformation parameters including the position information of the original scene element in the original live space and the scene matching parameters on the live terminal interface of the target live space.

[0064] Specifically, the user drags the original scene element in the live broadcast interface to place it at any position. When the dragging starts, a drag command is generated and the real-time moving position of the original scene element begins to be recorded. When the dragging ends, the final position information of the original scene element is recorded to generate the drag trajectory of the original scene element and obtain the corresponding spatial transformation parameters.

[0065] Specifically, the user selects the desired scene type in the live broadcast interface, selects the target element of the scene type from various original scene elements, and then generates scene matching information based on the preset position relationship of each type of original scene elements, such as the relative position relationship between the sofa and the coffee table, the relative position relationship between the TV and the TV cabinet, etc.; then, based on the initial position information of each original scene element and the scene matching information, the relative position relationship between each identified target element is determined, and combined with the initial position information of the live broadcast space where it is located, the corresponding space transformation parameters are generated.

[0066] 204. According to the live broadcast scene parameters corresponding to the live broadcast space, coordinately adjust the space effect of the original scene elements to obtain final scene elements;

[0067] In this embodiment, the target live broadcast space can be selected from the aforementioned live broadcast space or not, and the coordinated adjustment method of the spatial effect of the original scene elements adopted here is different. For the former, the coordinated adjustment of the spatial effect of more dimensions is performed on the original scene elements, as shown below:

[0068] 4.11) When the live broadcast space includes a target live broadcast space and other live broadcast spaces except the target live broadcast space, extracting the second camera parameter and the third scene rendering information from the live broadcast scene parameters corresponding to the target live broadcast space, and extracting the third camera parameter and the fourth scene rendering information from the live broadcast scene parameters corresponding to the other live broadcast spaces;

[0069] 4.12) constructing a mapping relationship between the target live broadcast space and the other live broadcast spaces according to the second camera parameter and the third camera parameter, and scaling the original scene elements according to the mapping relationship;

[0070] 4.13) determining a second space transformation parameter between the target live broadcast space and the other live broadcast space, and adjusting the position of the scaled original scene elements in the target live broadcast space according to the second space transformation parameter, and placing them according to the corresponding live broadcast terminal type;

[0071] 4.14) Eliminating the rendering effect of the original scene element after the adaptation and placement according to the fourth scene rendering information, and mapping the third scene rendering information to the original scene element after eliminating the rendering effect, to obtain the final scene element.

[0072] In this embodiment, the camera parameters (first camera parameter, second camera parameter and third camera parameter) include the focal length in the live broadcast space, the second camera parameter includes the focal length in the target live broadcast space, and the third camera parameter includes the focal length in other live broadcast spaces. There is no need to adjust the size of the original scene elements in the target live broadcast space; for the original scene elements in other live broadcast spaces, the mapping relationship between the original scene elements in the live broadcast space and the actual space is calculated according to the focal length of other live broadcast spaces, and the actual size of the original scene elements is calculated according to the size mapping relationship, and then the mapping relationship between the original scene elements in the actual space and the target live broadcast space is calculated according to the focal length in the target live broadcast space, so as to calculate the size of the original scene elements in other live broadcast spaces in the target live broadcast space, and complete the size adjustment of the original scene elements.

[0073] Specifically, two different types of second space transformation parameters are generated in the previous step. One of the second space transformation parameters is set according to the requirements to place the original scene elements of other live broadcast spaces to the adjusted positions in the target live broadcast space, and to perform adaptive placement adjustment according to the different terminal types between the other live broadcast spaces and the target live broadcast space, as shown below:

[0074] According to the first space transformation parameter, the position of the final scene element is adjusted in the target live broadcast space, and a first live broadcast terminal type corresponding to the target live broadcast space and a second live broadcast terminal type corresponding to the live broadcast space are identified;

[0075] Obtaining a first interface size corresponding to the first live broadcast terminal type and a second interface size corresponding to the second live broadcast terminal type;

[0076] An interface mapping relationship between the first interface size and the second interface size is calculated, and the scene elements after position adjustment are placed in the target live broadcast space according to the interface mapping relationship.

[0077] In this embodiment, for example, the second live broadcast terminal type of other live broadcast spaces includes tablet computers and laptop computers, and the first live broadcast terminal type of the target live broadcast space is a smart phone. Then, according to the interface mapping relationship between the second interface sizes of the tablet computers and laptop computers and the first interface sizes of the smart phone, the adaptation placement strategy is pre-set. Here, the original scene elements can be directly placed from the smart phone to the interfaces of the tablet computer and laptop computer respectively according to the adaptation placement strategy.

[0078] In this embodiment, the scene rendering information in the original scene elements (first scene rendering information, second scene rendering information, third scene rendering information and fourth scene rendering information) may include rendering effects such as light, saturation, and color, and the fourth scene rendering information in other live broadcast spaces is eliminated, so that each original scene element is an object that only has a spatial physical structure.

[0079] Then, the fourth scene rendering information of the target live broadcast space is mapped to the selected original scene elements, and the color, saturation, light, etc. of the original scene elements are re-rendered to obtain the final scene elements. The brightness and direction of the light are adjusted, and the cast shadows of the original scene elements are also adjusted to make them coordinated in the overall target live broadcast space.

[0080] It should be noted that for each live broadcast space, there is a corresponding set of different camera parameters and scene rendering information.

[0081] If the target live broadcast space is not selected from the aforementioned live broadcast space, a coordinated adjustment of more dimensional space effects is performed on the original scene elements, which can be specifically as follows:

[0082] 4.21) extracting first camera parameters and first scene rendering information from the live scene parameters corresponding to the live space;

[0083] 4.22) Calculate the actual size of the original scene element according to the first camera parameter, scale the original scene element to the actual size, and perform rendering effect culling on the original scene element according to the first scene rendering information to obtain a final scene element.

[0084] In this embodiment, if the target live broadcast space is not determined here, all the original scene elements are directly coordinated and adjusted to obtain the final scene elements of a unified style. Since the target live broadcast space is not determined. The final scene elements here only perform the elimination of scene rendering information and the calculation of actual size. That is, according to the first photography parameter, the focal length of the live broadcast space is determined to calculate the mapping relationship between the original scene elements in the live broadcast space and the actual space, and then according to the mapping relationship, the original scene elements are proportionally reduced or enlarged to obtain the original scene elements in actual size. At the same time, the rendering effect of the first scene rendering information is eliminated to obtain a scene element with only the physical structure of the space after proportional scaling.

[0085] 205. Synthesize the final scene element with the real-time live content of the target live space to obtain mixed live content, and broadcast the mixed live content in the target live space.

[0086] In this embodiment, for the target live broadcast space to be selected from the aforementioned live broadcast space or not to be selected from the aforementioned live broadcast space, a corresponding solution is adopted to perform collaborative adjustment of the spatial effect on the original scene element, and here, for different collaborative adjustment methods, corresponding synthesis methods of the scene element and the real-time live broadcast content are also performed. For the first collaborative adjustment method mentioned above, a first synthesis method is provided, which is specifically as follows:

[0087] 5.11) Determine second scene rendering information corresponding to the target live broadcast space, and first space transformation parameters between the target live broadcast space and the target live broadcast space;

[0088] 5.12) According to the first space transformation parameter, the final scene element is adjusted in position in the target live broadcast space, and is placed according to the corresponding live broadcast terminal type, and the second scene rendering information is mapped to the scene element after the placement of the adaptation;

[0089] 5.13) Acquire each video frame corresponding to the real-time live content of the target live space, and perform time alignment between the mapped scene elements and each video frame;

[0090] 5.14) According to the result of the time alignment, the final scene element is synthesized with each video frame to obtain a new video frame, and a mixed live content is generated based on the new video frame.

[0091] In this embodiment, the position adjustment, adaptive placement and re-rendering of scene elements in the second collaborative adjustment method are performed here, and the specific process is not repeated. Later, the scene elements and the video frames of the target live broadcast space are synthesized at the same time to obtain new video frames to generate mixed live broadcast content. In the mixed live broadcast content, the real-time live broadcast content of the target live broadcast space and the scene elements of other live broadcast spaces acquired at the same time are broadcast simultaneously.

[0092] In view of the second collaborative adjustment method mentioned above, a second synthesis method is provided, as shown below:

[0093] 5.21) Determine each video frame corresponding to the real-time live content of the target live space, and perform time alignment between the final scene element and each video frame;

[0094] 5.22) According to the result of the time alignment, the final scene element is synthesized with each video frame to obtain a new video frame, and a mixed live content is generated based on the new video frame.

[0095] In this embodiment, after the second collaborative adjustment method is completed for the original scene elements, they can be directly used for the synthesis of mixed live content. Figure 3 As shown, in the synthesis process of the target live broadcast space and the scene elements of other live broadcast spaces, if live broadcast space 1 is used as the target live broadcast space, after selecting the scene elements (scene elements in the dotted box) in live broadcast space 1 and live broadcast space 2 for generating mixed live broadcast content, the unselected scene elements in live broadcast space 1 are eliminated, and the current video frame is merged with the scene elements of live broadcast space 2 to form a new video frame for generating mixed live broadcast content and broadcasting it to the target live broadcast space.

[0096] The hybrid live broadcast method in the embodiment of the present invention is described above. The hybrid live broadcast device in the embodiment of the present invention is described below. Figure 4 In one embodiment of the present invention, a hybrid live broadcast device includes:

[0097] The capture module 401 is used to obtain the real-time live content of the live broadcast space to be broadcasted, and to capture the original scene elements in the real-time live content;

[0098] An adjustment module 402 is used to coordinately adjust the space effect of the original scene elements according to the live scene parameters corresponding to the live space to obtain the final scene elements;

[0099] The synthesis module 403 is used to synthesize the final scene element with the real-time live content of the target live space to obtain mixed live content, and broadcast the mixed live content in the target live space.

[0100] In the embodiment of the present invention, the original scene elements are captured by capturing the real-time live content in different live spaces, such as the shooting content of multiple live rooms or multiple cameras; each original scene element contains the environment rendering data under the corresponding live space, or the actual size scaling degree caused by different focal lengths, so each original scene element is uniformly and collaboratively adjusted for spatial effects to obtain standard scene elements, which are synthesized with the real-time live content of the target live space to be broadcast, and the final scene elements are integrated into the real-time live content, and then rebroadcast in the target live space. This allows the same live page to display the contents of two physical spaces at the same time, and the elements of the real and virtual spaces can achieve mixed live broadcasting.

[0101] See also Figure 5 Another embodiment of the hybrid live broadcast device in the embodiment of the present invention includes:

[0102] The capture module 401 is used to obtain the real-time live content of the live broadcast space to be broadcasted, and to capture the original scene elements in the real-time live content;

[0103] An adjustment module 402 is used to coordinately adjust the space effect of the original scene elements according to the live scene parameters corresponding to the live space to obtain the final scene elements;

[0104] The synthesis module 403 is used to synthesize the final scene element with the real-time live content of the target live space to obtain mixed live content, and broadcast the mixed live content in the target live space.

[0105] Specifically, the adjustment module 402 includes a first processing unit 4021, which is used to:

[0106] Extracting first camera parameters and first scene rendering information from the live scene parameters corresponding to the live space;

[0107] According to the first camera parameter, the actual size of the original scene element is calculated, the original scene element is scaled to the actual size, and the rendering effect of the original scene element is culled according to the first scene rendering information to obtain a final scene element.

[0108] Specifically, the synthesis module includes a first synthesis unit 4031, which is used to:

[0109] Determine second scene rendering information corresponding to the target live broadcast space and first space transformation parameters between the target live broadcast space and the target live broadcast space;

[0110] According to the first space transformation parameter, the final scene element is adjusted in position in the target live broadcast space, and is placed according to the corresponding live broadcast terminal type, and the second scene rendering information is mapped to the scene element after the placement of the adaptation;

[0111] Acquire each video frame corresponding to the real-time live content of the target live space, and perform time alignment between the mapped scene elements and the each video frame;

[0112] According to the result of the time alignment, the final scene element is synthesized with each video frame to obtain a new video frame, and mixed live content is generated based on the new video frame.

[0113] Specifically, the first synthesis unit 4031 is further used for:

[0114] According to the first space transformation parameter, the position of the final scene element is adjusted in the target live broadcast space, and a first live broadcast terminal type corresponding to the target live broadcast space and a second live broadcast terminal type corresponding to the live broadcast space are identified;

[0115] Obtaining a first interface size corresponding to the first live broadcast terminal type and a second interface size corresponding to the second live broadcast terminal type;

[0116] An interface mapping relationship between the first interface size and the second interface size is calculated, and the scene elements after position adjustment are placed in the target live broadcast space according to the interface mapping relationship.

[0117] Specifically, the adjustment module further includes a second processing unit 4022, which is used to:

[0118] When the live broadcast space includes a target live broadcast space and other live broadcast spaces except the target live broadcast space, extracting the second camera parameter and the third scene rendering information from the live broadcast scene parameters corresponding to the target live broadcast space, and extracting the third camera parameter and the fourth scene rendering information from the live broadcast scene parameters corresponding to the other live broadcast spaces;

[0119] According to the second camera parameter and the third camera parameter, a mapping relationship between the target live broadcast space and the other live broadcast spaces is constructed, and the original scene element is scaled according to the mapping relationship;

[0120] Determine a second space transformation parameter between the target live broadcast space and the other live broadcast space, and adjust the position of the scaled original scene elements in the target live broadcast space according to the second space transformation parameter, and adapt and place them according to the corresponding live broadcast terminal type;

[0121] The original scene elements after adaptation and placement are subjected to rendering effect elimination according to the fourth scene rendering information, and the third scene rendering information is mapped to the original scene elements after the rendering effect is eliminated, so as to obtain the final scene elements.

[0122] Specifically, the synthesis module further includes a second synthesis unit 4032, which is used to:

[0123] Determine each video frame corresponding to the real-time live content of the target live space, and perform time alignment between the final scene element and each video frame;

[0124] According to the result of the time alignment, the final scene element is synthesized with each video frame to obtain a new video frame, and mixed live content is generated based on the new video frame.

[0125] Specifically, the hybrid live broadcast device further includes a trajectory recognition module 404, which is used to:

[0126] In response to the dragging instructions of the original scene elements on the live terminal interface, respectively, determining the dragging trajectory of the original scene elements, and generating corresponding space transformation parameters according to the dragging trajectory; and / or,

[0127] Identify the scene matching information of the original scene element on the live terminal interface, and generate corresponding space transformation parameters based on the initial position information of the original scene element and the scene matching information;

[0128] The spatial transformation parameters include the first spatial transformation parameters and the second spatial transformation parameters.

[0129] Specifically, the capture module includes an identification capture unit 4011, which is used to:

[0130] Respectively identifying interface elements in each of the real-time live broadcast contents, and intercepting each identified interface element;

[0131] In response to a selection instruction of a selection button corresponding to each of the interface elements on the live broadcast terminal interface, the target interface element is captured and used as an original scene element in each of the real-time live broadcast contents.

[0132] Specifically, the capture module further includes a clipping capture unit 4012, which is used to:

[0133] In response to the cropping instructions for each of the real-time live content on the live terminal interface, determine the cropping range for each of the real-time live content;

[0134] The local real-time live broadcast content within the cropping range is used as the original scene element in each of the real-time live broadcast contents.

[0135] above Figure 4 and Figure 5 The hybrid live broadcast device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the hybrid live broadcast device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0136] Figure 6 : is a structural schematic diagram of a hybrid live broadcast device provided by an embodiment of the present invention. The hybrid live broadcast device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the hybrid live broadcast device 600. Furthermore, the processor 610 can be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the hybrid live broadcast device 600.

[0137] The hybrid live broadcast device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating systems 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The structure of the hybrid live broadcast device shown does not constitute a limitation of the hybrid live broadcast device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0138] The present invention also provides a hybrid live broadcast device, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the hybrid live broadcast method in the above-mentioned embodiments.

[0139] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the hybrid live broadcast method.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0142] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid live broadcast method, characterized in that: The hybrid live broadcast method comprises: Acquire the real-time live broadcast content of the live broadcast space to be broadcasted, and capture the original scene elements in the real-time live broadcast content; According to the live scene parameters corresponding to the live space, the original scene elements are collaboratively adjusted in terms of spatial effects to obtain final scene elements; The final scene element is synthesized with the real-time live content of the target live space to obtain mixed live content, and the mixed live content is broadcasted in the target live space; The collaborative adjustment of the space effect of the original scene elements according to the live scene parameters corresponding to the live space to obtain the final scene elements includes: When the live broadcast space includes a target live broadcast space and other live broadcast spaces except the target live broadcast space, extracting the second camera parameter and the third scene rendering information from the live broadcast scene parameters corresponding to the target live broadcast space, and extracting the third camera parameter and the fourth scene rendering information from the live broadcast scene parameters corresponding to the other live broadcast spaces; According to the second camera parameter and the third camera parameter, a mapping relationship between the target live broadcast space and the other live broadcast spaces is constructed, and the original scene element is scaled according to the mapping relationship; Determine a second space transformation parameter between the target live broadcast space and the other live broadcast space, and adjust the position of the scaled original scene elements in the target live broadcast space according to the second space transformation parameter, and adapt and place them according to the corresponding live broadcast terminal type; The original scene elements after adaptation and placement are subjected to rendering effect elimination according to the fourth scene rendering information, and the third scene rendering information is mapped to the original scene elements after the rendering effect is eliminated, so as to obtain the final scene elements.

2. The hybrid live broadcast method according to claim 1, characterized in that: The collaborative adjustment of the space effect of the original scene elements according to the live scene parameters corresponding to the live space to obtain the final scene elements includes: Extracting first camera parameters and first scene rendering information from the live scene parameters corresponding to the live space; According to the first camera parameter, the actual size of the original scene element is calculated, the original scene element is scaled to the actual size, and the rendering effect of the original scene element is culled according to the first scene rendering information to obtain a final scene element.

3. The hybrid live broadcast method according to claim 2, characterized in that: The step of synthesizing the final scene element with the real-time live content of the target live space to obtain mixed live content includes: Determine second scene rendering information corresponding to the target live broadcast space and first space transformation parameters between the target live broadcast space and the target live broadcast space; According to the first space transformation parameter, the final scene element is adjusted in position in the target live broadcast space, and is placed according to the corresponding live broadcast terminal type, and the second scene rendering information is mapped to the scene element after the placement of the adaptation; Acquire each video frame corresponding to the real-time live content of the target live space, and perform time alignment between the mapped scene elements and the each video frame; According to the result of the time alignment, the final scene element is synthesized with each video frame to obtain a new video frame, and mixed live content is generated based on the new video frame.

4. The hybrid live broadcast method according to claim 3, characterized in that: The step of adjusting the position of the final scene element in the target live broadcast space according to the first space transformation parameter, and placing the final scene element according to the corresponding live broadcast terminal type includes: According to the first space transformation parameter, the position of the final scene element is adjusted in the target live broadcast space, and a first live broadcast terminal type corresponding to the target live broadcast space and a second live broadcast terminal type corresponding to the live broadcast space are identified; Obtaining a first interface size corresponding to the first live broadcast terminal type and a second interface size corresponding to the second live broadcast terminal type; An interface mapping relationship between the first interface size and the second interface size is calculated, and the scene elements after position adjustment are placed in the target live broadcast space according to the interface mapping relationship.

5. The hybrid live broadcast method according to claim 1, characterized in that: The step of synthesizing the final scene element with the real-time live content of the target live space to obtain mixed live content includes: Determine each video frame corresponding to the real-time live content of the target live space, and perform time alignment between the final scene element and each video frame; According to the result of the time alignment, the final scene element is synthesized with each video frame to obtain a new video frame, and mixed live content is generated based on the new video frame.

6. The hybrid live broadcast method according to any one of claims 3 to 5, characterized in that: Before the spatial effect of the original scene elements is collaboratively adjusted according to the live scene parameters corresponding to the live space to obtain the final scene elements, the method further includes: In response to the dragging instructions of the original scene elements on the live terminal interface, respectively, determining the dragging trajectory of the original scene elements, and generating corresponding space transformation parameters according to the dragging trajectory; and / or, The scene matching information of the original scene element on the live terminal interface is identified, and the initial position information of the original scene element and the scene matching information are used to generate corresponding space transformation parameters.

7. The hybrid live broadcast method according to claim 1, characterized in that: The capturing of the original scene elements in the real-time live broadcast content includes: Respectively identifying interface elements in each of the real-time live broadcast contents, and intercepting each identified interface element; In response to a selection instruction of a selection button corresponding to each of the interface elements on the live broadcast terminal interface, the target interface element is captured and used as an original scene element in each of the real-time live broadcast contents.

8. The hybrid live broadcast method according to claim 1, characterized in that: The capturing of the original scene elements in the real-time live broadcast content includes: In response to the cropping instructions for each of the real-time live content on the live terminal interface, determine the cropping range for each of the real-time live content; The local real-time live broadcast content within the cropping range is used as the original scene element in each of the real-time live broadcast contents.

9. A hybrid live broadcast device, characterized in that: The hybrid live broadcast device comprises: A capture module, used to obtain the real-time live content of the live broadcast space to be broadcasted, and to capture the original scene elements in the real-time live content; An adjustment module, configured to coordinately adjust the spatial effects of the original scene elements according to the live scene parameters corresponding to the live space, to obtain final scene elements; A synthesis module, used for synthesizing the final scene element with the real-time live content of the target live space to obtain mixed live content, and broadcasting the mixed live content in the target live space; The adjustment module is specifically used to extract the second camera parameter and the third scene rendering information from the live scene parameters corresponding to the target live space, and to extract the third camera parameter and the fourth scene rendering information from the live scene parameters corresponding to the other live spaces when the live broadcast space includes the target live broadcast space and other live broadcast spaces except the target live broadcast space; construct a mapping relationship between the target live broadcast space and the other live broadcast spaces according to the second camera parameter and the third camera parameter, and scale the original scene elements according to the mapping relationship; determine the second space transformation parameter between the target live broadcast space and the other live broadcast space, and adjust the position of the scaled original scene elements in the target live broadcast space according to the second space transformation parameter, and adapt and place them according to the corresponding live broadcast terminal type; perform rendering effect removal on the adapted and placed original scene elements according to the fourth scene rendering information, and map the third scene rendering information to the original scene elements after the rendering effect is removed to obtain the final scene elements.

10. A hybrid live broadcast device, characterized in that: The hybrid live broadcast device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the hybrid live broadcast device executes the various steps of the hybrid live broadcast method as described in any one of claims 1-8.

11. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the hybrid live broadcast method as described in any one of claims 1-8 are implemented.

Images