Multi-video device acquisition and display method and system

Abstract

The application provides a multi-video device acquisition and display method and system. The method comprises the following steps: a user interacts with video input source data obtained by a second process through a user interaction module in a first process, the first process obtains a user instruction; the first process sends the user instruction to the second process; the first process establishes a drawing area according to the user instruction and maps the drawing area to a window of the second process; the second process calls video input source data according to the user instruction, processes the video input source data, and displays the processed video input source data in the window. The low-latency multi-video device acquisition and display method and system support simultaneous acquisition and display of multiple video source devices, greatly improve the video experience, and meet the needs of different users and scenes.

Description

Technical Field

[0001] This invention relates to the field of video communication technology, and more specifically, to a method and system for acquiring and displaying video from multiple devices. Background Technology

[0002] Internet-based video communication technology is widely used in video conferencing scenarios for both work and life. Among these technologies, multi-video device capture and display based on Android devices plays a crucial role in video conferencing, providing users with a richer video communication experience, flexible layout and switching functions, enhanced video effects, and remote monitoring capabilities. This enables participants to communicate and share information in real time, resulting in a richer and more intuitive video conferencing experience.

[0003] In the prior art, video acquisition and display systems based on Android devices operate in a single process, which includes the following modules: UI (user interaction module), Surface (drawing area), Window (window), and video source device.

[0004] The UI is the module in an Android device responsible for handling the user interface, managing, controlling, and interacting with it, such as displaying the camera preview interface and displaying player controls.

[0005] Surface is a graphics drawing area in an Android device, responsible for graphics rendering and display operations, managing and compositing all graphics layers, and rendering them onto the screen. Surface can be thought of as a canvas on which graphics can be drawn or image data can be rendered.

[0006] A window is the visible area of ​​a user interface.

[0007] The existing acquisition and display process operates as follows: the UI receives user instructions, such as start preview, end preview, etc., and requests the video device to acquire video. The Surface obtains video texture data processed by graphics rendering technology such as OpenGL ES (OpenGL for Embedded Systems). There is a mapping relationship between the Surface and the Window. The video texture data in the Surface will be further rendered onto the screen area corresponding to the Window to realize the display of video images.

[0008] The textured video data mentioned is not the original video data. Texture data refers to image data used in graphics rendering. In graphics programming, a texture typically refers to a two-dimensional image used for mapping, texture mapping, or image rendering. Texture data can be generated from underlying data and is usually stored in specific formats and specifications, such as texture objects in OpenGL ES. During the graphics rendering process, texture data is mapped onto the surface of a 3D model to achieve mapping effects or other visual effects.

[0009] Traditional Android devices face bottlenecks when handling multiple video display devices and multiple video source devices. They cannot simultaneously enable multiple devices for display or process data from multiple video sources concurrently. For example, processing data from video input source 'a' cannot process data from video input source 'b', and vice versa. Furthermore, the long display chain on Android devices results in significant latency in displaying the captured video data; the average latency on Android phones is greater than 200ms, and on high-quality Android TVs, it is greater than 300ms. Since Surface can only obtain rendered texture data and cannot manipulate underlying data, new videos require re-rendering and processing after the user resets parameters, resulting in delayed display and further latency. Summary of the Invention

[0010] This invention provides a dual-process multi-video device acquisition and display method and system, which directly retrieves and processes raw video source data, supports simultaneous acquisition and display of multiple video source devices, achieves multiple inputs and multiple outputs, and has very low latency, reaching about 130ms, which greatly improves the video experience and meets the needs of different users and scenarios.

[0011] In a first aspect, the present invention provides a method for capturing and displaying multiple video devices, characterized in that the system for capturing and displaying multiple video devices includes a first process of a first operating system and a second process of a second operating system, the method comprising: The user interacts with the video input source data obtained by the second process through the user interaction module in the first process, and the first process receives user instructions; The first process sends the user instruction to the second process; The first process creates a drawing area according to the user's instructions and maps it to the window of the second process; The second process retrieves video input source data according to the user's instructions, processes the video input source data, and displays the processed video input source data in the window.

[0012] Secondly, the present invention also provides a multi-video device acquisition and display system, characterized in that the system includes a first process of a first operating system and a second process of a second operating system, the first process including a user interaction module, a sending module, an establishment module, and a mapping module, and the second process including a processing module and a display module, wherein: The user interaction module is used for the user to interact with the video input source data obtained by the second process, so that the first process can obtain user instructions; The sending module is used to send the user instruction to the second process; The creation module is used to create a drawing area according to the user instructions; The mapping module is used to map the drawing area to the window of the second process; The processing module is used to retrieve video input source data according to the user instruction and process the video input source data; The display module is used to display the processed video input source data in the window.

[0013] The multi-video device acquisition and display system provided by this invention includes a first process running an Android operating system and a second process running a Linux operating system, i.e., two processes from different operating systems. The second process can directly process the raw video source data, realizing on-demand retrieval of the raw video source data, i.e., the underlying data, with very low latency, reaching about 130ms, which greatly improves the video experience. Furthermore, it can support simultaneous acquisition and display of multiple video source devices, with multiple inputs and multiple outputs. This invention can simultaneously acquire multiple video inputs, and the second process can simultaneously operate on multiple input source data, meeting the user's need to display any video source data on any video output device. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is an interactive diagram of the multi-video device acquisition and display process provided in an embodiment of the present invention; Figure 2 This is a flowchart of the multi-video device acquisition and display method provided in an embodiment of the present invention; Figure 3 This is the graphical interface of the user interaction module provided in the embodiments of the present invention; Figure 4This invention provides a graphical interface for displaying multiple local video input source data at the output end, as provided in an embodiment of the invention. Figure 5 This is a block diagram of a multi-video device acquisition and display system provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Invention Overview As mentioned above, the present invention provides a low-latency multi-video device acquisition and display method and system, which significantly improves the video experience and meets the needs of different users and scenarios.

[0018] Exemplary methods like Figure 1 As shown, the multi-video device acquisition and display system provided in this invention includes a first process of a first operating system and a second process of a second operating system. The first operating system is an Android operating system, and the second operating system is a Linux operating system. The Linux kernel provides low-level hardware drivers, process management, memory management, and device abstraction layer functions.

[0019] The second process is responsible for manipulating the underlying data, i.e., the raw data, while the first process cannot directly manipulate the raw data. Because the two processes reside on different operating systems, they cannot be merged.

[0020] The first process and the second process communicate via Binder. Binder communication is a mechanism in Android devices used for inter-process communication (IPC) between processes operating different operating systems. Through Binder communication, different processes can share data and call remote methods, enabling functional sharing and collaboration. Because Binder communication is implemented at the kernel level, it has high performance and security, and is widely used for interaction between Android applications and system services.

[0021] Figure 2 This is a flowchart of a multi-video device acquisition and display method provided in an embodiment of the present invention. This embodiment includes the following steps: S101: The user interacts with the video input source data obtained by the second process through the user interaction module (UI) in the first process, and the first process obtains user instructions.

[0022] The video input source data refers to input data from one or more video devices. This means that the user can interact with multiple video source data simultaneously through the user interaction module, such as controlling the start, stop, and camera switching of the video devices. Users can control and adjust the video devices by touching the screen or using the UI elements provided by the system. Input sources for the video devices include, but are not limited to, cameras, HDMI, and screen recording.

[0023] The user commands include, but are not limited to: start preview, stop preview, set preview size, get input source number, set input terminal number, and set output terminal number.

[0024] For example, if user A clicks on the user interaction module to preview all video input source data a~e of the second process, and selects to view the video data of input sources a, c, and d on output terminal 1 (e.g., a projector), then the first process receives the user command to start previewing, obtain the input source number, set the input terminal numbers a, c, and d, and set the output terminal number 1. Output terminal devices include, but are not limited to, projectors, monitors, mobile phone screens, computer screens, electronic whiteboards, in-vehicle display terminals, and television equipment.

[0025] S102: The first process sends the user instruction to the render manager of the second process.

[0026] Specifically, the first process sends the user command to the second process via Binder communication.

[0027] S103: The first process creates a drawing area (Surface) according to the user instruction and maps it to the window (Window) of the second process.

[0028] In Android, Surface is an abstract concept used for image rendering and display. It can be viewed as a producer of image data, containing image data from multiple input sources, such as camera preview data and video decoding data. Surfaces can be created in different ways, such as through SurfaceView, TextureView, and MediaCodec.

[0029] Window is an abstract concept in the Linux operating system that represents an application window and is used to render image data onto the screen.

[0030] In Android devices, there is a mapping relationship between Surface and Window for the display and rendering of video data: Surface can be associated with one or more Windows; when Surface is associated with Window, the image data in Surface is rendered onto the screen area corresponding to Window; by associating Surface with a specific Window, video data can be displayed at a specified location in the application's user interface; different Windows can be associated with different Surfaces, thereby enabling the display of different image data in different locations or on different screens.

[0031] When the user selects multiple output terminal numbers, the first process can create multiple drawing areas based on the output terminal numbers and map them to multiple windows of the second process.

[0032] For example, if user B chooses to view video data from input sources a, c, and d in three different windows numbered 1, 2, and 3 at the output end, the first process creates three drawing areas according to the user's instructions and maps them to the three windows of the second process respectively.

[0033] For example, if user C chooses to view video data of different resolutions from input source a in four windows numbered 1, 2, 3, and 4 at the output end, then the first process creates four drawing areas according to the user's instructions and maps them to the four windows of the second process respectively.

[0034] When the user selects an output terminal number, the first process can create a drawing area based on the output terminal number and map it to a window of the second process.

[0035] For example, such as Figure 3 As shown, user D selects a window on output terminal 1 in the user interaction module to view video data from the input source HDMI input and the built-in camera. The first process then creates a drawing area based on the user's instructions and maps it to a window in the second process, displaying the result as shown. Figure 4 As shown.

[0036] When the user selects one output terminal number, the first process can create multiple drawing areas based on the output terminal number and map them to multiple windows of the second process.

[0037] For example, if user E chooses to view the video data of input sources a, b, c, and d in the four windows of output terminal 1 respectively, then the first process creates four drawing areas according to the user's instructions and maps them to the four windows of the second process respectively.

[0038] The specific steps of the first process establishing a drawing area according to the user instruction and mapping it to the window of the second process are as follows: The first process creates the drawing area according to the user instruction, and passes the drawing area to the second process through the user instruction interface. The second process then obtains the window through the drawing area.

[0039] S104: The rendering manager of the second process retrieves the video input source data according to the user instruction, processes the video input source data, and displays the processed video input source data in the window.

[0040] The video input source data mentioned above is the raw data, also known as the original video source data. Raw data typically refers to the original, unprocessed image or video data. It can be pixel-level image data or video frame data, represented in raw binary form. Raw data can be stored in different formats and encoding methods, such as RGB, YUV, ARGB, etc. During processing, raw data often requires decoding, processing, and encoding operations to facilitate subsequent image processing, compression, or display.

[0041] Because the second process in the Linux operating system can directly call and process the underlying data, avoiding the encoding and decoding process of the video input source data, the video can be processed and displayed faster, and the overhead of the Android operating system itself is saved. Therefore, this invention has a lower latency than the prior art, with a latency of about 130ms for a 1080p30 video input source, which greatly improves the user's video experience.

[0042] Specifically, the second process uses V4L2 (Video4Linux2) to perform video acquisition, video output, video processing, and device control on the video input source data.

[0043] In the Linux operating system, V4L2 (Video4Linux2) is a framework for video device drivers, used to support operations such as video capture, video output, video processing, and device control. It provides APIs and interfaces for interacting with video devices (such as cameras).

[0044] V4L2 offers many features and functionalities, including: Video capture: V4L2 allows applications to capture real-time video data from video devices such as cameras. It provides functions for configuring device parameters, starting and stopping capture, and acquiring capture frames.

[0045] Video Output: V4L2 supports sending video data to video output devices, such as monitors or video encoders. It provides functions for configuring output devices, setting video formats, and sending video frames.

[0046] Video processing: V4L2 allows for various processing of video streams, such as image enhancement, filter application, and color space conversion. It provides a series of control interfaces and operation functions for processing video data.

[0047] Device control: V4L2 provides control functions for video devices, such as adjusting camera parameters like exposure, focus, and white balance. It also supports querying device capabilities, such as supporting video formats and resolutions.

[0048] By using the V4L2 framework, developers can write applications on Linux systems to control and operate video devices. It is a widely used video driver framework in Linux systems, providing applications with interfaces to access and manipulate video devices.

[0049] The second process, in particular, includes the following steps: retrieving video input source data according to the user instruction and processing the video input source data: First, the video input source data is converted to obtain the converted data, such as by decoding, processing and encoding the underlying data; The converted data is then rendered to obtain rendered data. For example, SurfaceView can be used to render the decoded video frame data to display continuous video images; by binding video textures to OpenGL ES textures, OpenGL ES can be used for 3D graphics rendering, such as image enhancement, filter application, and real-time effects, to render video frames onto the screen and achieve more complex video effects and transitions.

[0050] Unlike existing technologies that can only process one video input source at a time, this invention supports simultaneous acquisition and display of multiple video source devices. It can process multiple video input source data simultaneously and display them in one or more windows according to user needs, i.e., multiple inputs and multiple outputs. The second process can directly manipulate any video input source data, and the first process can arbitrarily switch drawing areas. Therefore, it allows for any combination of multiple video input source data and display windows, with different windows appearing on a single display screen or on different screens. For example, multiple windows can display different video input source data; multiple screens can be split-screen to display different or the same video input source data. Therefore, this invention supports arbitrary input and output settings in video conferencing, meeting the needs of different users in different scenarios.

[0051] For example, the processed video input source data displayed in the same window may come from different video input sources, which can be achieved through methods such as window splitting, window overlay, texture mapping, and image compositing.

[0052] The split-window feature divides the window into multiple regions, displaying one video data point in each region. Multiple SurfaceViews or TextureViews can be created as needed, and each video data point can be associated with its corresponding view. This allows each video data point to be displayed in an independent region, achieving the effect of playing multiple videos simultaneously.

[0053] An overlay window displays multiple video data sets superimposed on a single window. A SurfaceView or TextureView can be used as the underlying window, and multiple video data sets can be drawn on this window using Canvas or OpenGL ES. By controlling properties such as the drawing position, size, and transparency of the video data, the overlay and display effects can be achieved.

[0054] Texture mapping involves mapping multiple video data sets as textures onto a 3D model surface for display. By using OpenGL ES for texture mapping, each video data set can be bound to a texture object, which is then mapped onto a shared 3D model surface. This allows multiple video data sets to be displayed within the same window at different positions and angles.

[0055] Image compositing involves combining multiple video data points into a single image and displaying that image in a window. Image processing libraries or graphics programming techniques can be used to combine multiple video frames according to certain rules, generating a composite image containing multiple video data points. This composite image is then displayed in a window, achieving the effect of displaying multiple video data points.

[0056] For example, the processed video input source data displayed in different windows comes from the same video input source. User F needs to display the video data of input sources a, c, and d simultaneously in two windows of different sizes. The second process directly retrieves the 1080P and 360P video data of input sources a, c, and d according to the user's instructions and processes them.

[0057] In summary, the second process in this invention can operate multiple input source data and multiple output devices simultaneously, solving the problem that the prior art cannot process multiple video source data at the same time, and meeting the user's need for arbitrary video source data to be displayed on any video output device.

[0058] Exemplary System Accordingly, embodiments of the present invention also provide a multi-video device acquisition and display system. Figure 5 This is a block diagram of a multi-video device acquisition and display system provided in an embodiment of the present invention, such as... Figure 5 As shown, the system 100 provided in this embodiment includes: A first process 101 of a first operating system and a second process 102 of a second operating system, wherein the first process 101 includes a user interaction module 103, a sending module 104, an establishment module 105, and a mapping module 106, and the first process 102 includes a processing module 107 and a display module 108, wherein: The user interaction module 103 is used for the user to interact with the video input source data obtained by the first process 102, so that the first process 101 can obtain user instructions. The sending module 104 is used to send the user instruction to the first process 102; The creation module 105 is used to create a drawing area according to the user instructions; The mapping module 106 is used to map the drawing area to the window of the first process 102; The processing module 107 is used to retrieve video input source data according to the user instruction and process the video input source data; The display module 108 is used to display the processed video input source data in the window.

[0059] The video input source data is the original video source data.

[0060] The video input source data comes from multiple video devices.

[0061] The user instructions specifically include: start preview, stop preview, set preview size, get input source number, set input terminal number, and set output terminal number.

[0062] When there are multiple output terminal numbers, the establishment module 105 is used to establish multiple drawing regions according to the output terminal numbers, and the mapping module 106 is used to map the multiple drawing regions to multiple windows of the first process 102.

[0063] The processing module 107 includes a conversion submodule 109 and a rendering submodule 110, wherein: The conversion submodule 109 is used to convert the video input source data to obtain the converted data; The rendering submodule 110 is used to render the converted data to obtain the rendered data.

[0064] The mapping module 106 includes a transmission submodule 111, wherein the establishment module 105 is used to establish the drawing area according to the user instruction, and the transmission submodule 111 is used to transmit the drawing area to the first process 102 through the user instruction interface, and the first process 102 obtains the window through the drawing area.

[0065] The first process 101 and the first process 102 communicate via Binder.

[0066] The processed video input source data displayed in the same window comes from different video input sources.

[0067] The processed video input source data displayed in the different windows comes from the same video input source.

[0068] The first operating system is Android, and the second operating system is Linux.

[0069] The processing module 107 is used to perform video acquisition, video output, video processing, and device control on the video input source data via V4L2 (Video4Linux2).

[0070] It should be noted that although the operation of the multi-video device acquisition and display method of the present invention is described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0071] Furthermore, although several devices, units, or modules of a multi-video device acquisition and display system have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0072] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for acquiring and displaying video from multiple devices, characterized in that, The method for providing a multi-video device acquisition and display system includes a first process of a first operating system and a second process of a second operating system, and the method includes: The user interacts with the video input source data obtained by the second process through the user interaction module in the first process, and the first process receives user instructions; The first process sends the user instruction to the second process; The first process creates a drawing area according to the user's instructions and maps it to the window of the second process; The second process retrieves video input source data according to the user's instructions, processes the video input source data, and displays the processed video input source data in the window; The video input source data is the original video source data; The video input source data is input source data from multiple video devices; The user instructions specifically include: start preview, stop preview, set preview size, get input source number, set input terminal number, and set output terminal number; The method further includes: When there are multiple output terminal numbers, the first process establishes multiple drawing areas according to the output terminal numbers and maps them to multiple windows of the second process; The first operating system is Android, and the second operating system is Linux.

2. The multi-video device acquisition and display method according to claim 1, characterized in that, The second process, in particular, includes the following steps: retrieving video input source data according to the user instruction and processing the video input source data: The video input source data is converted to obtain the converted data; The transformed data is then rendered to obtain the rendered data.

3. The multi-video device acquisition and display method according to claim 1 or 2, characterized in that, The specific steps of the first process establishing a drawing area according to the user instruction and mapping it to the window of the second process are as follows: The first process creates the drawing area according to the user instruction, and passes the drawing area to the second process through the user instruction interface. The second process then obtains the window through the drawing area.

4. The multi-video device acquisition and display method according to claim 1 or 2, characterized in that, The first process and the second process communicate via Binder.

5. The multi-video device acquisition and display method according to claim 1 or 2, characterized in that, The processed video input source data displayed in the same window comes from different video input sources.

6. The multi-video device acquisition and display method according to claim 1 or 2, characterized in that, The processed video input source data displayed in the different windows comes from the same video input source.

7. The multi-video device acquisition and display method according to claim 1 or 2, characterized in that, The second process retrieves video input source data according to the user instruction and processes the video input source data as follows: The second process performs video acquisition, video output, video processing, and device control on the video input source data through V4L2.

8. A multi-video device acquisition and display system, characterized in that, The system includes a first process of a first operating system and a second process of a second operating system. The first process includes a user interaction module, a sending module, an establishment module, and a mapping module. The second process includes a processing module and a display module, wherein: The user interaction module is used for the user to interact with the video input source data obtained by the second process, so that the first process can obtain user instructions; The sending module is used to send the user instruction to the second process; The creation module is used to create a drawing area according to the user instructions; The mapping module is used to map the drawing area to the window of the second process; The processing module is used to retrieve video input source data according to the user instruction and process the video input source data; The display module is used to display the processed video input source data in the window; The video input source data is the original video source data; The video input source data is input source data from multiple video devices; The user instructions specifically include: start preview, stop preview, set preview size, get input source number, set input terminal number, and set output terminal number; When there are multiple output terminal numbers, the establishment module is used to establish multiple drawing regions according to the output terminal numbers, and the mapping module is used to map the multiple drawing regions to multiple windows of the second process; The first operating system is Android, and the second operating system is Linux.

9. The multi-video device acquisition and display system according to claim 8, characterized in that, The processing module includes a conversion submodule and a rendering submodule, wherein: The conversion submodule is used to convert the video input source data to obtain the converted data; The rendering submodule is used to render the converted data to obtain the rendered data.

10. The multi-video device acquisition and display system according to claim 8 or 9, characterized in that, The mapping module includes a transmission submodule, wherein the establishment module is used to establish the drawing area according to the user instruction, and the transmission submodule is used to transmit the drawing area to the second process through the user instruction interface, and the second process obtains the window through the drawing area.

11. The multi-video device acquisition and display system according to claim 8 or 9, characterized in that, The first process and the second process communicate via Binder.

12. The multi-video device acquisition and display system according to claim 8 or 9, characterized in that, The processed video input source data displayed in the same window comes from different video input sources.

13. The multi-video device acquisition and display system according to claim 8 or 9, characterized in that, The processed video input source data displayed in the different windows comes from the same video input source.

14. The multi-video device acquisition and display system according to claim 8 or 9, characterized in that, The processing module is used to perform video acquisition, video output, video processing, and device control on the video input source data via V4L2.

Images