Multi-path live broadcast picture synthesis system and method based on virtual camera
Through virtual camera and layer management technology, the problem of insufficient personalization and flexibility in the existing live broadcast technology is solved, and the custom synthesis of multiple live broadcast images is realized, which improves the real-time and smoothness of live broadcasts and simplifies equipment management , significantly improving the live broadcast experience and resource utilization efficiency.
Patent Information
- Application Number
- CN202411498225.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing live broadcast technology solution is difficult to achieve personalized and flexible picture customization in multiple live broadcast scenarios, with poor real-time and smoothness, and it is difficult to flexibly manage the live broadcast requirements of multiple devices and multiple scenarios, resulting in complex device configuration and debugging. .
Adopt a multi-channel live picture synthesis system based on virtual cameras. Through layer management technology and virtual camera technology, highly customized synthesis of pictures is achieved, hardware dependence is reduced, real-time and smoothness are improved by using GPU rendering technology, and a variety of live content is supported. integration.
It realizes highly customized live broadcast effects, reduces hardware dependence, improves system flexibility and real-timeness, simplifies device configuration and management, and significantly improves live broadcast experience and resource utilization efficiency.
Smart Images

Figure CN119094673A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video processing, and in particular to a multi-channel live broadcast picture synthesis system and method based on a virtual camera. Background Art
[0002] Typical live broadcast technology solutions on the current market, such as open broadcast software OBS (Open Broadcaster Software), mainly rely on physical cameras to capture live content and use software to perform simple editing and synthesis of images. These solutions can achieve basic live image processing, but they have great defects in image customization in various live broadcast scenarios, resulting in the inability to perform personalized and flexible settings. For example: Lack of personalization: Existing technologies mainly use preset templates and special effects to achieve image processing, which makes it difficult to highly customize according to different live broadcast scenarios and user needs.
[0003] Poor real-time and smoothness: Complex image synthesis and special effects processing often lead to large consumption of system resources, thus affecting the real-time and smoothness of live broadcast.
[0004] Limited flexibility: Under the requirements of live broadcasting on multiple devices and in multiple scenarios, existing technologies cannot flexibly switch and manage image data from different sources.
[0005] In summary, existing live broadcast technology solutions usually install multiple hardware equipment acquisition devices in the environment to capture the screen of the live broadcast host. Due to equipment limitations, such hardware can usually only output content in the real world. But in fact, the live broadcast screen may need to display product pictures, text introductions, or other recorded pictures at the same time, but the existing live broadcast system cannot provide highly customized synthesis of multiple pictures, and requires multiple external physical devices to cooperate to complete it. It is difficult to debug and affects the live broadcast experience. At the same time, OBS needs to manually configure each video source when processing multiple pictures, which is complicated and error-prone. In addition, the adjustment of parameters such as the level, position, and angle of the picture is cumbersome, and there is a lack of personalized, automated, and intelligent management mechanisms. Summary of the invention
[0006] A multi-channel live broadcast screen synthesis system and method based on virtual camera is proposed, which can be used in the field of live broadcast and can provide highly customized live broadcast effects. In the live broadcast scene, the system and application interface are converted into virtual camera screens through software simulation camera, and the technology is used as the bottom-level live broadcast screen output; layer management technology is used to arrange and set the parameters such as the level, position, angle, display size, transparency, etc. of multi-channel live broadcast screens to generate personalized live broadcast screens; through multi-channel screen synthesis technology, the virtual camera is connected to the physical camera, and the screen of the physical camera can be synthesized as a layer of the virtual camera, and the virtual background, special effects and digital human elements are integrated to synthesize the screens into the same screen.
[0007] The multi-channel live broadcast picture synthesis system based on virtual cameras includes the following modules: The camera driver module uses a virtual camera driver that complies with the Linux video device driver V4L2 standard interface specification to receive the original graphics buffer provided by the user; The picture synthesis module receives the image stream, provides a graphical user interface GUI program to control the picture synthesis, and outputs the synthesized image buffer; The synthetic image display module is connected to the camera driver module and the image synthesis module, maps the synthetic graphics buffer produced by the image synthesis module to an external texture, adjusts the image viewing angle through matrix transformation, produces the final graphics buffer, writes it to the memory pointer specified by the camera driver module, and notifies the camera driver module; In the composite image display module, the hardware abstraction layer HAL is re-implemented; an open graphics library OpenGL environment is created in the HAL to simulate the acquired image and realize the virtual camera. The built-in OpenGL environment is controlled through the newly added virtual camera hardware abstraction layer VirtualCameraHAL, including controlling the addition and deletion of virtual cameras and adjusting the rendering parameters of the virtual cameras.
[0008] The method for synthesizing multiple live broadcast images based on a virtual camera includes the following steps: Step 1, receiving the original graphics buffer provided by the user; Step 2, for the received image stream, synthesize it into a graphics buffer for output; Step 3, mapping the synthesized graphics buffer to an external texture and binding it to the OpenGL environment; Step 4: Perform matrix transformation on the vertices through the vertex shader to adjust the image viewing angle, achieve the effects of rotation, stretching, displacement, and scaling, and produce the final graphics buffer; then notify VirtualCameraHAL to consume the graphics buffer through cross-process communication; Step 5: Output the final synthesized graphics buffer for display.
[0009] The beneficial effects achieved by the present invention are: (1) Highly customized live broadcast effects: The present invention combines virtual camera technology with a layer management module to enable users to perform highly customized synthesis of multiple images. Users can freely define and adjust the layer, position, angle, display size, transparency and other parameters of the image, thereby achieving personalization and specialization of the live broadcast image.
[0010] (2) Reduce hardware dependence: Since the present invention supports the synthesis of virtual cameras and multi-channel images, users no longer need to rely on multiple external physical devices to display different content. This not only simplifies the configuration and management of live broadcast equipment, but also reduces the complexity of the debugging process and improves the flexibility and convenience of the overall system.
[0011] (3) Improved real-time performance and fluency: The present invention adopts optimized graphics processor GPU rendering technology and virtual camera technology. The entire rendering process is completed completely on the GPU side, without the need for excessive involvement of the central processing unit CPU. During the entire process, the graphics data is copied 0 times, and no data readback is required, which significantly reduces system resource consumption and improves the efficiency and real-time performance of image processing. Even in complex image synthesis scenarios, the system can still maintain efficient operation and ensure the smoothness and stability of the live broadcast process.
[0012] (4) Multifunctional integration: This system can not only display the real-time captured host screen, but also integrate various contents such as product display, text introduction and pre-recorded video. Through the layer management module, users can easily synthesize these contents into the same screen to achieve rich live broadcast effects.
[0013] (5) Significantly improve the live broadcast experience: By providing highly customized live broadcast images, the present invention greatly improves the professionalism and viewing quality of live broadcasts. Users can flexibly adjust and optimize live broadcast content according to different live broadcast scenes and needs, thereby providing a better viewing experience.
[0014] (6) Simplify equipment configuration and management: The invention reduces the reliance on multiple external physical devices and simplifies the configuration and management process of live broadcast equipment. Users only need to set and adjust through software to achieve complex picture synthesis, which reduces the difficulty and cost of equipment debugging.
[0015] In the past, content live broadcasting scenes required at least two or more devices to complete multi-channel picture synthesis, but the present invention only requires one live broadcast machine to complete it. The present invention supports more than 10 types of live picture synthesis, including video, camera, picture, HDMI, screen projection, digital human, document, screen mirroring, text, etc.
[0016] (7) Improve system flexibility and adaptability: The present invention supports the free selection and definition of multiple channels of images, and can adapt to the complex live broadcast requirements of multiple devices and multiple scenes. Whether it is a simple live broadcast of a single scene or a complex live broadcast of multiple scenes and multiple contents, the system can flexibly respond and provide stable and high-quality live broadcast services.
[0017] (8) Optimizing resource utilization: By optimizing the picture processing and synthesis algorithm, the present invention improves the utilization efficiency of system resources, reduces hardware requirements and resource consumption. Even in a high-load live broadcast environment, the system can still run stably to ensure the smooth progress of the live broadcast process. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a detailed architecture diagram of a multi-channel live broadcast picture synthesis system based on a virtual camera in an embodiment of the present invention; Figure 2 A schematic diagram of a camera creation process in an embodiment of the present invention; Figure 3 A schematic diagram of a process of opening a virtual camera by a third-party application in an embodiment of the present invention; Figure 4 A schematic diagram of image production in an embodiment of the present invention; Figure 5 Schematic diagram of a system user interface in an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.
[0020] The system provided by the present invention comprises: The camera driver module is a virtual camera driver that fully complies with the Linux video device driver V4L2 standard interface specification. Different from the physical camera driver, the data source comes from the hardware sensor. This module provides device nodes to the user space, receives the original graphics buffer from the user space, and returns it to the standard driver caller, that is, the AndroidCamera framework that comes with Android.
[0021] Image synthesis module: Provides a complete graphical user interface GUI program to control image synthesis, which can receive any other single or multiple image streams (such as images, videos, cameras, or a third-party application, or any Android standard control), and synthesize them into a graphics buffer through the skia2D engine.
[0022] Synthetic picture display module: connect the camera driver module with the picture synthesis module, map the graphic buffer produced by the picture synthesis module to an external texture through the built-in open graphics library OpenGL environment, and bind it to the OpenGL environment of this module. Perform matrix transformation on the vertices through the vertex shader to achieve the effect of adjusting the viewing angle (rotation, stretching, displacement, scaling, etc.), produce the final graphic buffer, and notify the virtual camera hardware abstraction layer VirtualCameraHAL module.
[0023] The screen synthesis module also supports direct output of any Android application screen, and supports running the delivered application in an independent running environment. By creating a virtual screen in the Android Display subsystem and connecting the graphics buffer of this virtual screen to the screen synthesis module, the screen can be output to the camera driver. Next, the composite screen delivery module starts the application to be delivered on the virtual screen through the activity manager service AMS (ActivityManagerService) interface to achieve the purpose of independent operation, similar to the form of an independent small window, which can greatly improve the efficiency of the use of the physical screen. This module can also provide simulation of any number of virtual cameras, custom resolution size, and selection of supported delivery / output graphics buffer formats.
[0024] like Figure 1 As shown in the figure, the custom camera hardware abstraction layer CameraHAL module is used to implement the standard AndroidCamera HAL for the Android system to request the graphics buffer. The screen synthesis module is used to control the CPU and GPU to perform real-time rendering and synthesis for multiple graphics buffer sources according to the user's preferences, and establish a connection with the VirtualCameraHAL to pass the synthesized graphics buffer to the custom CameraHAL module.
[0025] The screen synthesis module can provide an open graphics library GUI program for users to manually adjust the synthesis effect and preview it, so as to meet the needs of users to build customized digital live studios. It also has the capabilities of layer rotation, layer rendering order, layer cutting, layer scaling and stretching. For example, users can place a specified layer in the front for rendering, or rotate it at any angle along the Z axis before rendering.
[0026] This system also includes customization for the Android CameraHAL part. By customizing and adjusting the camera preemption strategy of CameraHAL, users can open multiple cameras as input sources at the same time in this synthesis system. And finally, they are synthesized and rendered into a graphics buffer and transmitted to the camera driver module. Through the camera preemption strategy, multiple processes can open multiple cameras at the same time between different applications. By adjusting the maximum budget MaxCost value for each camera device in the Android Camera Server, it directly determines whether the corresponding camera device is in a busy state. By adjusting the MaxCost value of the virtual camera, it will be enlarged to a specific multiple of the original value, making it more difficult for the virtual camera to reach the busy state determined by Android, so as to achieve the purpose of performing specific release operations on the virtual camera's camera opening request.
[0027] In the composite image display module, the hardware abstraction layer HAL is re-implemented; an OpenGL environment is created in the HAL to simulate the acquired image and realize the virtual camera. The built-in OpenGL environment is controlled through the newly added VirtualCameraHAL, including controlling the addition and deletion of virtual cameras and adjusting the rendering parameters of the virtual cameras.
[0028] refer to Figure 2 , the creation process of the virtual camera is as follows: The newly added VirtualCameraHAL registers a virtual camera across processes. In VirtualCameraHAL, a separate OpenGL environment is created for this camera to render virtual images. When the environment is prepared, Android is notified of the access of a new Camera device through Android's standard CameraHAL. Then, applications developed through Android's standard interface can open the virtual camera just like opening a real physical camera.
[0029] After the virtual camera is created, there is no content on the screen by default. The content is customized by the user through the GUI program, that is, the screen is rendered by the user-side software; whether to start rendering depends on whether other applications call the virtual camera through the Android standard application programming interface API.
[0030] refer to Figure 3 , the process of the application opening the virtual camera and rendering the image is as follows: First, call the Android standard interface to apply to open the camera viewfinder; then CameraServer connects to the corresponding CameraHAL and determines whether it is a virtual camera ID; if so, the newly added VirtualCameraHAL returns the OpenGL image buffer, that is, the image of the virtual camera; otherwise, the Android standard CameraHAL returns the hardware image buffer, that is, the image of the physical camera.
[0031] refer to Figure 4 , image production will only start after a user requests a virtual camera, so its process corresponds to the process of opening a virtual camera and rendering an image, as follows: The Android app opens the camera through the Android standard camera framework, and the camera id is specified at this time. When the selected camera is consistent with the virtual camera app, the system will call the customized CameraHAL of this system, and the HAL will call back to notify the live assistant app. At this time, the live assistant app will request an OpenGL texture from the VirtualCameraHAL and render the image to it to complete the image production.
[0032] The system also includes an audio virtual module, which intercepts the audio output stream of the specified target application by adding an AudioPolicy audio policy, replaces it with the audio recording AudioRecord in this module, and copies the intercepted target audio raw stream to the audio track AudioTrack as the audio input of the camera driver module. For example, when the application is performing audio input or audio output, it needs to obtain the audio input / output interface through AudioPolicy. When returning the corresponding audio input / output interface, it will match the package name of the caller and return the newly added AudioPolicy that meets the rules to the caller. The audio input / output interface of this AudioPolicy points to the virtual audio module, so the purpose of audio redirection is achieved, and the virtual audio module can freely process the data stream transmitted by the application through the audio input / output interface.
[0033] Reference Figure 5 , this system enables users to highly customize the synthesis of multiple images through the combination of virtual camera technology and layer management module. Users can freely define and adjust the layer, position, angle, display size, transparency and other parameters of the image, so as to achieve the personalization and specialization of the live broadcast image. In addition, this system can not only display the host image collected in real time, but also integrate various contents such as product display, text introduction and pre-recorded video. Through the layer management module, users can easily synthesize these contents into the same image to achieve rich live broadcast effects.
[0034] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A multi-channel live broadcast picture synthesis system based on virtual cameras, characterized by: The system includes the following modules: The camera driver module uses a virtual camera driver that complies with the Linux video device driver V4L2 standard interface specification to receive the original graphics buffer provided by the user; The picture synthesis module receives the image stream, provides a graphical user interface GUI program to control the picture synthesis, and outputs the synthesized image buffer; The synthetic image display module is connected to the camera driver module and the image synthesis module, maps the synthetic graphics buffer produced by the image synthesis module to an external texture, adjusts the image viewing angle through matrix transformation, produces the final graphics buffer, writes it to the memory pointer specified by the camera driver module, and notifies the camera driver module; In the composite image display module, the hardware abstraction layer HAL is re-implemented; an open graphics library OpenGL environment is created in the HAL to simulate the acquired image and realize the virtual camera. The built-in OpenGL environment is controlled through the newly added virtual camera hardware abstraction layer VirtualCameraHAL, including controlling the addition and deletion of virtual cameras and adjusting the rendering parameters of the virtual cameras.
2. The multi-channel live broadcast picture synthesis system based on virtual camera according to claim 1, characterized in that: The system also includes an audio virtual module, which adds an audio policy AudioPolicy through the system API. The policy filters the package names of specified applications and redirects the audio output and audio input of these applications.
3. The multi-channel live broadcast picture synthesis system based on virtual camera according to claim 1, characterized in that: Through the camera preemption strategy, multiple processes can open multiple cameras simultaneously between different applications. By adjusting the Android Camera Server, there is a maximum budget MaxCost value for each camera device, which directly determines whether the corresponding camera device is in a busy state. By adjusting the MaxCost value of the virtual camera, it is enlarged to the corresponding multiple of the original value according to demand, making it more difficult for the virtual camera to reach the busy state determined by Android, so as to achieve the purpose of performing specific release operations on the virtual camera's camera opening request.
4. The multi-channel live broadcast picture synthesis system based on virtual camera according to claim 1, characterized in that: The image synthesis module receives any other single or multiple image streams, including images, videos, cameras, third-party applications, and any Android standard controls.
5. The multi-channel live picture synthesis system based on virtual camera according to claim 1, characterized in that: The screen synthesis module uses the skia2D engine to synthesize single or multiple image streams into a graphics buffer.
6. The multi-channel live broadcast picture synthesis system based on virtual camera according to claim 1, characterized in that: The composite image display module realizes the mapping of the graphics buffer through the built-in OpenGL environment, and performs matrix transformation on the vertices through the vertex shader to achieve the viewing angle adjustment effect of rotation, stretching, displacement and scaling.
7. The multi-channel live broadcast picture synthesis system based on virtual camera according to claim 1, characterized in that: The screen synthesis module creates a virtual screen in the Android Display subsystem and connects the graphics buffer of this virtual screen to the screen synthesis module to achieve the purpose of outputting this screen to the camera driver. Next, the synthetic screen sending module starts the application to be delivered on the virtual screen through the activity manager service AMS interface to achieve the purpose of independent operation.
8. The multi-channel live broadcast picture synthesis system based on virtual camera according to claim 1, characterized in that: The process of creating a virtual camera is as follows: When the software on the user side is started, it will interact with the newly added VirtualCameraHAL through cross-process communication to register a virtual camera. In VirtualCameraHAL, a separate OpenGL environment is created for this camera to render the virtual image. When the environment is prepared, Android is notified of the access of a new camera device through Android's standard camera hardware abstraction layer CameraHAL. At this time, third-party applications directly open the virtual camera through the Android standard interface. At this time, third-party applications cannot perceive whether this camera is a real camera or a virtual camera. After the virtual camera is created, there is no content in the screen by default. The content is customized by the user through the GUI program, that is, the screen is rendered through the screen synthesis module; whether to start rendering depends on whether other applications call the virtual camera through the Android standard application program interface API; image production will only start after the third-party application requests the virtual camera through the Android standard API; The process of opening the virtual camera and rendering the image is as follows: First, call the Android standard interface to apply to open the camera viewfinder; then the camera service CameraServer connects to the corresponding CameraHAL and determines whether it is a virtual camera ID; if so, the newly added VirtualCameraHAL returns the OpenGL image buffer, that is, the image of the virtual camera; otherwise, the Android standard CameraHAL returns the hardware image buffer, that is, the image of the physical camera.
9. A method for synthesizing a multi-channel live broadcast screen synthesis system based on a virtual camera according to any one of claims 1 to 8, characterized in that: The method comprises the following steps: Step 1, receiving the original graphics buffer provided by the user; Step 2, for the received image stream, synthesize it into a graphics buffer for output; Step 3, mapping the synthesized graphics buffer to an external texture and binding it to the OpenGL environment; Step 4: Perform matrix transformation on the vertices through the vertex shader to adjust the image viewing angle, achieve the effects of rotation, stretching, displacement, and scaling, and produce the final graphics buffer; then notify VirtualCameraHAL to consume the graphics buffer through cross-process communication; Step 5: Output the final synthesized graphics buffer for display.
Citation Information
Patent Citations
Image display method and device based on virtual camera, storage medium and electronic equipment
CN112804459A
Image or video data acquisition method and terminal equipment
CN113220445A
Method for displaying third-party application interface
CN116627577A
Live broadcast data processing method and device, equipment and storage medium
CN116708867A
Virtual director system and virtual director method
CN117411979A
