A third-party application interface display method

Abstract

The present invention discloses a third-party application interface display method, which is applied to a live broadcast machine. The present invention changes the interface display path of the third-party application based on the Android system architecture, so that the third-party application draws its screen information on a virtual display screen, while the electronic display screen of the live broadcast machine directly displays only the interface of the main control application. The interface of the third-party application is displayed through the main control application interface, which solves the conflict problem when multiple third-party applications are displayed at the same time, and facilitates the main control application to control the third-party application interface; further, the image acquisition interface and the audio interface are directly called through the main control application to share audio data and image data with the third-party application, thereby enriching the image resources and audio resources of the live broadcast application and improving the user experience of the live broadcast.

Description

Technical Field

[0001] The present invention relates to the field of Android system live broadcast machines, and in particular to a third-party application interface display method. Background Art

[0002] In the Android system, the current multi-window operation of multiple third-party applications on the same screen is achieved based on the multi-window technologies provided by the Android system architecture, such as freeform and split-screen, that is, the main control application acts as a launcher to interact with third-party applications.

[0003] In this way, users can open multiple third-party applications on the main application interface, but the third-party application display windows are parallel to the main application. The main application cannot control the drawing frame rate and resolution dpi of the third-party application. Since the third-party application runs in a sub-window, it cannot run at the specified resolution. For example, if the physical screen resolution is 1080x1920 and the sub-window resolution of the third-party application is 576x1080, the clarity of the third-party application display will be reduced. In addition, not all third-party applications support split-screen multi-window technology. Third-party applications that do not support split-screen technology will not be able to display in split screen or the application interface control layout will be misaligned and abnormal in size. The main application cannot flexibly and efficiently obtain the third-party application's screen and perform some processing through the GPU, such as special effect filters, beauty, keying, scaling and cropping, etc.

[0004] Under the current Android system architecture, both the main control application and third-party applications have direct access to physical hardware resources, and both use the Android standard interface process to schedule the corresponding hardware resources according to the system-specified routing strategy. If the main control application and the third-party application need to simultaneously open image acquisition interfaces such as the Camera interface, UVC-in, SDI-in, and HDMI-in, or simultaneously open audio input interfaces such as the MIC, Headset, and HDMI-in, or audio output interfaces such as HDMI-out, Speaker, and Headphone, conflicts will arise. At the same time, because third-party applications access physical hardware resources independently, the main control application has no ability to control the hardware access path of third-party applications, such as specifying that audio data played by the third-party application be played through HDMI-out, Speaker, or both.

[0005] Currently, when the main control application shares image data (such as images in system storage or external storage cards, or image data input through HDMI-in or USB interfaces) with third-party applications, it is implemented at the kernel driver layer. The image data shared by the main control application cannot be effectively used in conjunction with the platform system's built-in physical camera interface acquisition.

[0006] In addition, when the main control application obtains the audio data output by the third-party application, it must be implemented through the Android standard audiorecord recording interface. During the process, a pop-up window will appear for user authorization. Not all third-party applications are compatible with this solution. In addition, when the audio data of the third-party application is synchronously played to the physical audio output interface, a copy is copied to the recording interface. The implementation is not efficient enough and is too dependent on actual physical hardware resources. Summary of the Invention

[0007] In view of this, because there is a conflict problem in the prior art when multiple third-party applications are running in multiple windows on the same screen, the present invention provides a third-party application interface display method, which is applied to a live broadcast machine. The live broadcast machine adopts the Android system, has a main control application and can run third-party applications. The live broadcast machine is provided with an electronic display screen, and the electronic display screen only displays the interface of the main control application. The interface of the main control application includes a third-party application interface display area. The method includes the following steps:

[0008] (1) The master control application creates a virtual display screen according to preset specific parameters, wherein the preset specific parameters include an ID corresponding to the virtual display screen;

[0009] (2) When a third-party application is triggered, the master application specifies a virtual display screen for it, and the third-party application draws the screen texture information of its interface on the corresponding virtual display screen according to the ID identification corresponding to the virtual display screen, and the master application obtains the screen texture information in the corresponding virtual display screen through the ID identification information;

[0010] (3) The main control application processes the image texture information of the third-party application and then draws it in the display area of ​​the third-party application interface;

[0011] (4) The master control application monitors the interface trigger event and updates the third-party application interface display area according to the trigger event.

[0012] In step (4), the main control application monitors the interface triggering event. If it is determined that the third-party application interface display area is triggered, updating the third-party application interface display area includes the following steps:

[0013] The main control application forwards the corresponding trigger event to the third-party application through the virtual display screen, and the third-party application responds to the trigger event and updates its screen texture information on the virtual display screen;

[0014] The main control application updates the third-party application interface display area according to the updated screen texture information.

[0015] Furthermore, the live broadcast machine is also provided with an image acquisition interface; the display interface of the main control application also includes a setting area. In the initial state, the setting area displays the live broadcast setting layout, and the live broadcast setting layout includes scene thumbnails and scene addition keys.

[0016] Among them, the main control application monitors the interface trigger event. If it is determined that the scene addition key is triggered, the setting area switches to the scene creation layout. The scene creation layout includes the selection key of the image acquisition interface. After the main control application detects that the interface selection is completed and the confirmation event is triggered, the setting area switches to the scene editing layout.

[0017] Among them, the scene editing layout includes a scene editing effect area, a layer thumbnail area and an image editing button area. The layer thumbnail area displays the image data information input by the selected image acquisition interface respectively; the layers in the layer thumbnail area can be dragged to the scene editing effect area, and the image editing button area includes image editing operation keys, and the image editing operation keys can perform image editing operations on the layers in the scene editing effect area; after the scene editing is completed, the setting area returns to the live broadcast setting layout, and the edited scene is added to the scene thumbnail.

[0018] In step (4), the main control application monitors the interface trigger event. If it is determined that the scene thumbnail is triggered, updating the third-party application interface display area includes the following steps:

[0019] The main control application obtains the image data information corresponding to the scene thumbnail, and processes the image data information according to the image editing settings selected when the corresponding scene is created;

[0020] The main control application shares the processed image data information with the third-party application, and the third-party application draws the processed image data information on its corresponding virtual display screen;

[0021] The main control application extracts the image texture information on the virtual display screen and draws it in the third-party application interface display area.

[0022] The main control application shares the processed image data information with the third-party application, including the following steps:

[0023] Create a virtual camera hardware abstraction layer based on the Android system architecture, and establish an access path for the third-party application to access the virtual camera hardware abstraction layer;

[0024] The master control application transmits the processed image data information to the virtual camera hardware abstraction layer through an inter-process communication strategy;

[0025] The application layer of the third-party application obtains the image data information through the virtual camera hardware abstraction layer.

[0026] Furthermore, the live broadcast setting layout includes a volume setting key; when the main control application detects that the volume setting key is triggered, the setting area switches to the volume setting layout, which includes an input volume setting layout and an output volume setting layout.

[0027] The main control application and the third-party application share audio data according to the settings of input volume and output volume.

[0028] Furthermore, the third-party application interface display area can display multiple third-party applications, and the main control application creates multiple virtual display screens and specifies different virtual display screens for different third-party applications.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] The third-party application interface display method in the present invention is based on the Android system architecture. Without modifying the third-party application, it changes the interface display path of the third-party application, so that the third-party application draws its screen information on the virtual display screen, and only the interface of the main control application is directly displayed on the live broadcast machine. The interface of the third-party application is displayed in the main control application interface, which solves the conflict problem when multiple third-party applications are displayed at the same time, and facilitates the main control application to control the third-party application interface; further, the image acquisition interface and audio interface on the live broadcast machine are directly called by the main control application, which facilitates data sharing between the main control application and the third-party application; without changing the third-party application, the interaction of the live broadcast application is expanded, and the user experience of the live broadcast is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a structural diagram of the audio and video processing device of the present invention;

[0032] Figure 2 This is a schematic diagram of the main control application interface of the audio and video processing device in the present invention;

[0033] Figure 3 This is a schematic diagram of information interaction between the master control application and the virtual hardware abstraction layer in the present invention;

[0034] Figure 4 This is a diagram of the main configuration information of the virtual audio hardware abstraction layer;

[0035] Figure 5 This is a schematic diagram of the path for a third-party application to call the audio playback interface in Example 1;

[0036] Figure 6This is a schematic diagram of the implementation process of a third-party application calling an audio playback interface in Example 1;

[0037] Figure 7 This is a schematic diagram of the path for a third-party application to call the recording interface in Example 2;

[0038] Figure 8 This is a schematic diagram of the implementation process of a third-party application calling a recording interface in Example 2;

[0039] Figure 9 This is a schematic diagram of the path for a third-party application to call a camera in Example 3;

[0040] Figure 10 This is a schematic diagram of the implementation process of a third-party application calling a camera in Example 3;

[0041] Figure 11 This is a schematic diagram of the interaction of the main control application displaying a third-party application when the live broadcast machine is displayed in split screen in Example 4;

[0042] Figure 12 This is a schematic diagram of the interaction between the main control application and the third-party application when the live broadcast machine displays split screens in Example 4;

[0043] Figure 13 This is a schematic diagram of the main control application interface when the live broadcast machine enters unicast mode in Example 4;

[0044] Figure 14 This is a schematic diagram of the main control application interface when the live broadcast machine sets the application scenario and adds a layer in Example 4;

[0045] Figure 15 This is a schematic diagram of the main control application interface when the live broadcast machine sets the application scenario to edit the layer in Example 4;

[0046] Figure 16 This is a schematic diagram of the main control application interface when the live broadcast machine adds music in Example 4;

[0047] Figure 17 This is a schematic diagram of the main control application interface when the live broadcast machine sets the volume in Example 4;

[0048] Figure 18 This is a schematic diagram of the main control application interface when the live broadcast machine switches to multicast mode in Example 4;

[0049] Figure 19 This is a schematic diagram of the main control application interface when the live broadcast machine in Example 4 runs two third-party applications at the same time. Implementation Method

[0050] The present invention will be further described below in conjunction with the embodiments.

[0051] Figure 1The audio and video processing device includes a shell 1, in which a circuit board (not shown) is encapsulated, and an electronic display screen 2 is provided outside the shell 1. The shell 1 is provided with the following hardware: an HDMI-IN interface 3, an HDMI-OUT interface 4, a USB interface 5, an external memory card slot 6, a recording interface (Mic), an audio playback interface (Speaker) and a camera interface (not shown). The above-mentioned HDMI-IN interface 3, USB interface 5, and recording interface (Mic) serve as audio input interfaces, which can receive audio data; the above-mentioned HDMI-OUT interface 4, USB interface 5, and audio playback interface (Speaker) serve as audio output interfaces, which can output audio data; the above-mentioned audio input interface and audio output interface are collectively referred to as audio interfaces; the above-mentioned HDMI-IN interface 3, USB interface 5, and camera interface serve as image acquisition interfaces, which can acquire image data.

[0052] Figure 1 The audio and video processing devices in the Android system architecture are used, such as Figure 2 As shown, the system comes with a built-in main control application, and third-party applications can be installed, such as live broadcast applications and video conferencing applications. When these third-party applications are running on the device, they generally involve calling the interface display and audio and image acquisition interfaces. Commonly, playing audio requires calling the audio playback interface (Speaker), recording audio requires calling the recording interface (Mic), and capturing images requires calling the camera interface.

[0053] In the original Android system architecture, third-party applications and main control applications all use the same hardware calling path. The application layer accesses the hardware layer through the framework layer, hardware abstraction layer, and kernel layer in sequence to implement the calling of the hardware interface. The method of the present invention creates a virtual hardware abstraction layer based on the framework of the hardware abstraction layer in the Android system architecture, thereby changing the original path for third-party applications to call the device hardware interface.

[0054] The following will focus on the method of a third-party application calling a device hardware interface in the present invention, which includes the following steps:

[0055] (1) Create a virtual hardware abstraction layer and add it to the Android system source code for compilation to generate a dynamic library;

[0056] (2) When the system starts, the dynamic library corresponding to the virtual hardware abstraction layer is loaded, and an access path for third-party applications to access the virtual hardware abstraction layer is established;

[0057] (3) If Figure 3 As shown, when a third-party application calls a hardware interface, its application layer passes the call request to the framework layer, and the framework layer passes the call request to the virtual hardware abstraction layer;

[0058] (4) Based on the inter-process communication strategy, the virtual hardware abstraction layer passes the call request to the master application, and the master application transfers data between the virtual hardware abstraction layer according to the call request; the master application can access the hardware layer to directly call all hardware interfaces on the device, and pass the data received from the hardware interface to the virtual hardware abstraction layer, or pass the data received from the virtual hardware abstraction layer to the corresponding hardware interface.

[0059] Specifically, as described above, the hardware interfaces in the present invention include an audio interface and an image acquisition interface. A virtual audio hardware abstraction layer (AudioHAL) is created for the audio interface, and a virtual camera hardware abstraction layer (CameraHAL) is created for the image acquisition interface. For ease of understanding, the following describes how third-party applications use the present invention's method to obtain audio data from audio input interfaces such as the recording interface, transmit audio data to audio output interfaces such as the audio playback interface, and obtain image data from image acquisition interfaces such as the camera interface. Example 1

[0060] This embodiment takes a third-party application calling the audio playback interface Speaker to play audio data as an example to describe how the third-party application uses the hardware interface calling method of the present invention to call the audio playback interface to play audio data:

[0061] (1) Create a virtual audio hardware abstraction layer AudioHAL, add the virtual AudioHAL to the module module in the Android source code for compilation, and generate a dynamic library so after compilation is complete.

[0062] (2) Add the configuration information of the virtual audio hardware abstraction layer AudioHAL to the hardware policy configuration file audio_policy_configuration.xml, such as Figure 4 As shown, the configuration information includes: the module name modulename of the virtual AudioHAL, the version halVersion, the corresponding hardware resource attachedDevices, that is, the audio playback interface Speaker, the sampling rate SamplingRates of the input and output audio, the channel flag ChannelMasks and other information;

[0063] The framework layer parses the hardware resource configuration policy file to load the dynamic library information and establish an access path for the third-party application from the application layer, the framework layer to the virtual audio hardware abstraction layer AudioHAL.

[0064] (3) If Figure 5As shown in the figure, when a third-party application calls the audio playback interface to play audio data, the application layer calls Audiopolicy and Audioflinger in sequence through the AudioTrack interface in the framework layer Audio-framework, and then passes the call request to the virtual AudioHAL. The interaction process between the virtual audio hardware abstraction layer AudioHAL and the framework layer is consistent with the AudioHAL in the existing Android system. The main difference is that the virtual AudioHAL no longer accesses the kernel layer and finally accesses the hardware layer according to the original path.

[0065] (4) If Figure 3 As shown, the inter-process communication strategy in this embodiment is Unix domain socket, based on which JSON-IPC (inter-process communication) is implemented. The data interaction method of shared file descriptor information fd is adopted and JSON packaging and parsing are used to realize cross-process sharing of buf-fd and cross-process function calls.

[0066] The main control application obtains the call request of the third-party application through the cross-process function, determines the third-party application request to call the audio playback interface, and obtains the sampling rate, format, channel flag and other information of the virtual AudioHAL output audio through the cross-process function, and obtains the file descriptor information fd corresponding to the audio data to be played from the virtual AudioHAL, and obtains the third-party audio data from the memory according to the file descriptor information fd; combined with Figure 6 As shown, the main control application obtains the corresponding audio data according to the file descriptor information fd, and then calls the device's audio playback interface through the ALSA playback interface standard to play the audio data of the third-party application. At this point, the audio data to be played by the third-party application is played through the main control application's call to the audio playback interface.

[0067] In this embodiment, since the main control application can obtain the audio data of the third-party application, the audio data of the third-party application can be processed by noise reduction, tuning, etc. and then the audio playback interface can be called to play it.

[0068] The existing common audio output interfaces are not limited to the HDMI-OUT interface, USB interface, and audio playback interface Speaker used by the audio and video processing device in the present invention, but also include headphones, Bluetooth headphones BT-A2DP, etc. Because the main control application can obtain the audio data of the third-party application, and the main control application can directly call all device hardware interfaces, it is possible to select at least one audio output interface on the device to output the audio data of the third-party application according to needs during specific implementation. The main control application uses the virtual AudioHAL created at the bottom layer to interact and share audio data with the third-party application. The audio data played by the third-party application is synthesized by the system audiofinger service and then flowed into the virtual AudioHAL. The main control application obtains the data played by the third-party application by sharing buf-fd zero-copy across processes. After obtaining the audio data, the main control application can also perform noise reduction, tuning and other processing, and then select the audio output interface on the device to play according to business needs.

[0069] As an implementation method, the main control application can also send SIG_STOP SIG_CONT signals through the Linux underlying technology to stop or continue the process of the third-party application to pause and resume the audio playback of an application, or call a private interface to set the UUID of the third-party application and mute / unmute the audio stream of the corresponding third-party application without affecting its internal timestamp synchronization logic. It can also use the Android activity-manager related API to pause or resume the activity of the third-party app to pause and resume the audio playback of an app.

[0070] In summary, all audio output interfaces on the device are managed only by the master application, and third-party applications only recognize the virtual AudioHAL created in the present invention to play, while the actual audio playback is completed by the master application.

[0071] The above-mentioned HDMI-OUT interface, USB interface, speaker interface, and headphone are all applicable to ALSA interface. For the convenience of description, these interfaces are collectively referred to as first-class audio interfaces. The main control application is the application that comes with the audio and video processing device, and the system architecture modification is relatively flexible. For example, Figure 6As shown in the figure, in a specific implementation, the first type of audio interface can be called directly through the ALSA playback interface through the kernel layer driver, which can simplify the hardware interface calling process and reduce the call response time. However, for Bluetooth headsets BT-A2DP, this type of interface does not support the ALSA interface protocol. Therefore, when the main control application calls this type of interface, it follows the Android playback interface path, that is, calling through the application layer, framework layer, hardware abstraction layer, and kernel layer in sequence. For the sake of description, this type of interface is called the second type of audio interface. Example 2

[0072] This embodiment takes a third-party application calling the recording interface Mic to record audio data as an example to describe how the third-party application uses the hardware interface calling method of the present invention to call Mic to record audio data:

[0073] (1) Create a virtual audio hardware abstraction layer AudioHAL, add the virtual AudioHAL to the module module in the Android source code for compilation, and generate a dynamic library so after compilation. In the Android system architecture, the audio input interface and the audio output interface both use AudioHAL in the hardware abstraction layer, so this embodiment 2 can use the same virtual AudioHAL as embodiment 1.

[0074] (2) Add the configuration information of the virtual audio hardware abstraction layer AudioHAL to the hardware policy configuration file audio_policy_configuration.xml, such as Figure 4 As shown, the module name, version, input and output audio sampling rate SamplingRates, channel flag ChannelMasks and other information of the virtual AudioHAL, as well as the corresponding hardware resource attachedDevices, i.e. Built-In-Mic and other information;

[0075] The framework layer parses the hardware resource configuration policy file to load the dynamic library information and establish an access path for the third-party application from the application layer, the framework layer to the virtual audio hardware abstraction layer AudioHAL.

[0076] (3) If Figure 7 As shown in the figure, when a third-party application records audio data, the application layer calls the AudioRecord interface in the framework layer and then calls Audiopolicy and Audioflinger in sequence until the virtual AudioHAL passes the scheduling request. The interaction process between the virtual AudioHAL and the framework layer is consistent with the native AudioHAL in the existing Android system. The main difference is that the virtual AudioHAL no longer accesses the kernel layer according to the original path.

[0077] (4) If Figure 3 As shown, the inter-process communication strategy in this embodiment is the same as that in embodiment 1, that is, Unix domain socket is based on which JSON-IPC (inter-process communication) is implemented, a data interaction method of shared file descriptor information fd is adopted and json packaging and parsing is used to realize cross-process sharing of buf-fd and cross-process function calls.

[0078] like Figure 8 As shown, the main control application obtains the call request of the third-party application through the cross-process function, determines that the third-party application requests to call the recording interface, and sets the sampling rate, format, channel flag and other information of the virtual AudioHAL input audio through the cross-process function, and sends the file descriptor information fd corresponding to the recorded audio data to the virtual AudioHAL; the virtual AudioHAL obtains the corresponding audio data from the memory according to the file descriptor information fd, and combines Figure 7 As shown, the third-party application obtains the aforementioned audio data from the virtual AudioHAL through the Android standard recording interface. At this point, the third-party application records the audio data by calling the recording interface through the main control application.

[0079] As an implementation method, if the master application determines, based on a call request, that a third-party application is calling a recording interface, it selects at least one channel of audio data accessed by the audio input interface on the device and passes it to the virtual audio hardware abstraction layer. Existing common audio input interfaces are not limited to the HDMI-IN interface, USB interface, and recording interface Mic used by the audio and video processing device in the invention, but also include headset receivers, Bluetooth call BT-SCO, etc. Because the master application can directly call all device hardware interfaces, in a specific implementation, it can select audio data accessed by any one or more audio input interfaces on the device and pass it to the third-party application as needed. Since the master application directly calls each audio input interface, it can process the audio data accessed by each audio input interface through noise reduction, tuning, mixing, etc., and then send the processed data to the created virtual AudioHAL through a cross-process sharing buf-fd solution. When the third-party application calls the recording interface, it will obtain audio data from the virtual AudioHAL, thereby achieving sharing of input audio data.

[0080] In summary, all audio input interfaces on the device are managed only by the main control application, and third-party applications only recognize the virtual audio created by the present invention to record, while the actual audio recording is completed by the main control application.

[0081] For the HDMI-IN interface, USB interface, recording interface Mic, and headset, since they all apply ALSA interface, for the convenience of description, these interfaces are collectively referred to as the first-class audio interface. The main control application is the application that comes with the audio and video processing device, and the system architecture modification is relatively flexible, such as Figure 8 As shown in the figure, in a specific implementation, such hardware interfaces can be directly called through the ALSA recording interface through the kernel driver, which can simplify the hardware resource calling process and reduce the call response time. However, for Bluetooth headsets BT-SCO, such interfaces do not support the ALSA interface protocol. Therefore, when the host application calls such interfaces, it follows the Android playback interface path, that is, calling through the application layer, framework layer, hardware abstraction layer, and kernel layer in sequence. For ease of description, this interface is referred to as the second-class audio interface.

[0082] Obviously, the difference between Example 1 and Example 2 of the present invention is mainly based on the difference in the audio playback and recording paths themselves, and there is no essential difference in the applicability of the method of the present invention. Example 3

[0083] This embodiment takes a third-party application calling a camera interface to collect image data as an example to introduce how the third-party application uses the hardware interface calling method of the present invention to call a camera interface to collect image information.

[0084] (1) Create a virtual camera hardware abstraction layer CameraHAL, add the virtual CameraHAL to the module in the Android source code for compilation, and generate a virtual camera dynamic library so after compilation. Several virtual camera identifiers cam-id are registered in the virtual camera hardware abstraction layer. These virtual camera identifiers correspond one to one with virtual cameras. When multiple third-party applications call the camera at the same time, different virtual cameras are assigned to them, such as Figure 10 As shown, virtual camera 1 is assigned to the third-party application app1, and virtual camera 2 is assigned to the third-party application app2.

[0085] At the same time, the system camera hardware abstraction layer compiles and generates a dynamic library of actual cameras. The system camera hardware abstraction layer registers the actual camera identifier, which corresponds to the device's actual camera interface. The virtual camera identifier is different from the actual camera identifier. For example, in this embodiment, the virtual camera identifiers cam-id registered in the virtual CameraHAL are [0, 1, 2, 3, ..., 9], while the actual camera identifiers cam-id registered in the system camera hardware abstraction layer (shown in the figure as the system CameraHAL) are [10, 11, 12, 13, ...].

[0086] (2) This embodiment takes the HIDL mechanism as an example. Figure 9As shown, the first Camera.provider interface and the second Camera.provider interface are set. When the system starts, the CameraService process of the framework layer loads the virtual camera hardware abstraction layer CameraHAL according to the first Camera.provider interface, and establishes an access path for third-party applications from the application layer and the framework layer to the virtual CameraHAL; the CameraService process of the framework layer loads the system CameraHAL according to the second CameraProvider interface, and establishes an access path for the main control application from the application layer and the framework layer to the system CameraHAL.

[0087] The first CameraProvider interface loads the virtual camera hardware abstraction layer CameraHAL, and the second Camera.provider interface loads the system's own CameraHAL. The process is as follows: when Camera.provider is initialized, it loads the virtual camera dynamic library so through hw_module_get(), obtains the camera_module_t object and assigns it to CameraModule, and then obtains the virtual camera information through camera_module_t in CameraModule.

[0088] (3) When a third-party application calls the camera data, the application layer calls the Camera API and Cameraserver interface in the framework layer to call the first CameraProvider, and implements the call to the virtual CameraHAL through the first CameraProvider.

[0089] When the main control application calls the camera data, the application layer calls the Camera api and Cameraserver interface in the framework layer to call the second Camera.provider, and implements the call to the system CameraHAL through the second Camera.provider.

[0090] (4) Combination Figure 3 As shown, this embodiment adopts the same inter-process communication strategy as Example 1, namely, Unix domain socket json-rpc, adopts a data interaction method of sharing file descriptor information fd and uses json packaging and parsing to establish json-rpc, thereby realizing cross-process sharing of buf-fd and cross-process function calls.

[0091] The master application receives the call request from the third-party application through a cross-process function. Based on the call request, it determines whether the third-party application calls the camera interface and passes the image data collected by the actual camera interface to the virtual camera hardware abstraction layer. When the master application calls the actual camera interface, the application layer calls the Camera API and Cameraserver interface in the framework layer to call the second CameraProvider. Through the second CameraProvider, the system CameraHAL is called to call the actual camera.

[0092] like Figure 10 As shown, the main control application can obtain images captured by the actual camera interface through the Android system native path (Android Camera interface). The main control application obtains the virtual CameraHAL's description information such as format, width, height, and frame rate through a cross-process function, and sends the file descriptor information (fd) corresponding to the corresponding image data to the virtual CameraHAL. The virtual CameraHAL obtains the corresponding image data based on the file descriptor information (fd). At this point, the third-party application can obtain image data by calling the camera interface through the main control application. In this embodiment, the system CameraHAL refers to the chip of the corresponding module in the device.

[0093] As an implementation method, if the main control application determines that a third-party application calls the camera interface based on the call request, at least one of the image data captured by the image acquisition interface on the device is selected and passed to the virtual camera hardware abstraction layer; or the image data captured by the image acquisition interface on the device is edited and passed to the virtual camera hardware abstraction layer, and different image data can also be sent to different third-party applications (distinguished by virtual camera identifiers).

[0094] In this embodiment, Figure 10 As shown, in addition to the camera interface, the main control application can directly call the image acquisition interface such as the HDMI-IN interface and USB interface of the audio and video processing device. Since such interfaces are all applicable to the V4L2 interface, for the convenience of description, such interfaces are collectively referred to as the first type of image acquisition interface. The main control application is a built-in application of the audio and video processing device, and the system architecture modification is relatively flexible. In the specific implementation, the first type of image acquisition interface can be directly called through the V4L2 interface through the kernel layer driver, which can simplify the hardware resource call process and reduce the call response time. Of course, the V4L2 interface is also applicable to common image acquisition interfaces such as uvc-in and sdi-in, which belong to the first type of image acquisition interface. The camera interface in this embodiment does not support the V4L2 interface, so for the convenience of description it is called the second type of image acquisition interface. The main control application can only call the corresponding interface through the application layer, framework layer, hardware abstraction layer, and kernel layer in sequence as described above.

[0095] Because the master application can directly call the image acquisition interface on the device, when a third-party application calls the camera interface, the master application selects any image acquisition interface to capture image data and sends it to the third-party application. Alternatively, the master application can edit and process the image data captured by these interfaces and then pass it to the virtual CameraHAL via the file descriptor fd. Therefore, using the method of the present invention, the third-party application can not only obtain image data captured by the camera, but also obtain other image data shared by the master application.

[0096] In summary, the method of the third-party application calling the device hardware interface in the present invention is adopted to modify the path of the third-party application calling the audio interface and image acquisition interface. All audio interfaces and image acquisition interfaces are directly controlled by the main control application, which facilitates the unified management of audio data and image data and realizes data sharing between the main control application and the third-party application.

[0097] Based on the modification of the above Android system architecture, the audio and video processing device in the present invention can realize more functions at the business level. Example 4

[0098] In this implementation, the above-mentioned audio and video processing device is used as a live broadcast machine. The system of the live broadcast machine can install multiple third-party applications, such as live broadcast applications, to realize multicast function on the same host, and can process the image data required by each live broadcast application, and uniformly manage and control the audio data, so as to solve the conflict problems encountered by multiple applications jointly scheduling hardware interfaces and improve the user interaction experience.

[0099] The multiple third-party applications running in the live broadcast machine of this embodiment adopt the same hardware interface scheduling method as in embodiments 1, 2, and 3, so this will not be described in detail below. The focus will only be on the interface interaction between the third-party applications in this embodiment, especially the live broadcast application, and the main control application.

[0100] Combine Figure 11 and Figure 13 As shown, in this embodiment, the electronic display screen 2 only displays the various interfaces of the main control application. Taking the live broadcast interface of the main control application as an example, the third-party application is displayed through the main control application. The live broadcast interface of the main control application includes a third-party application interface display area. When the third-party application is running, the third-party application interface display area synchronously displays the display interface of the third-party application, including the following steps:

[0101] (1) The master application creates a virtual display screen according to preset specific parameters, where the preset specific parameters include the ID corresponding to the virtual display screen, pixel information such as width, height, and resolution;

[0102] (2) When a third-party application is triggered, the main control application specifies a virtual display screen for it. The third-party application draws the screen texture information of its interface on the corresponding virtual display screen according to the ID identification corresponding to the virtual display screen. The main control application obtains the screen texture information in the corresponding virtual display screen through the ID identification information;

[0103] (3) The main control application performs image processing on the image texture information, such as scaling, cropping, color balancing, beauty filters, etc., and then displays the processed image texture information in the third-party application interface display area to complete the interface display of the third-party application;

[0104] (4) The main control application monitors the interface trigger event and refreshes the third-party application interface display area according to the trigger event.

[0105] In the specific implementation, based on the Android system architecture design, the main control application can use the createVirtualDisplay function in DisplayManager to create multiple virtual displays with different specific parameters according to business needs, such as Figure 11 As shown, the physical display screen of the live broadcast machine is 1080x3840. The width and height of the virtual display screen 1 created by the main control application for the third-party application app1 are 1920x1080 according to the physical display screen, and the width and height of the virtual display screen 2 created for the third-party application app2 are 576x1080.

[0106] When triggering a third-party application, the main control application specifies that its activity start is in the ID identifier displayID corresponding to the corresponding virtual display. In this way, the third-party app will draw its layers according to the specified VirtualDisplay parameters such as width, height, dpi, etc. For example, third-party application app1 is drawn on virtual screen 1, and third-party application app2 is drawn on virtual screen 2.

[0107] In the Android system, the surface contains all the screen texture information used by third-party applications when drawing. After obtaining the surface information, the main control application can synchronously obtain the synthesized third-party application screen or GPU texture; the main control application draws the screen texture information extracted from virtual display screen 1 and virtual display screen 2 in the third-party application interface display area on the electronic display screen.

[0108] After acquiring the third-party application's screen texture, the control application can use the GPU to perform various processing, such as scaling, cropping, color balancing, applying beauty filters, and scene synthesis, before finally displaying it on the designated area of ​​the actual physical display screen according to business requirements. In short, the electronic display screen (the actual physical display) is managed solely by the control application component. Third-party applications run on the virtual screen and have no direct impact on the display's performance.

[0109] After the live broadcast machine is started, enter Figure 2 The main control application home page interface shown in the figure displays third-party application icons on the home page. Clicking any icon triggers the corresponding third-party application to start. Taking the third-party application as an example, after the live broadcast application is triggered, the system enters the live broadcast interface, including the following steps:

[0110] (1) The main control application determines that the live broadcast application (such as app1) is triggered and enters the live broadcast interface. The live broadcast interface layout includes the third-party application interface display area and the setting area. Figure 11 and Figure 13 As shown, the third-party application interface display area is synchronized with the display interface of the live broadcast application. The settings area displays the live broadcast setting layout, which includes existing scene thumbnails (such as Scene 1, Scene 2, Scene 3), a scene add button (+), and other setting buttons related to application functions such as music and volume;

[0111] (2) The main control application monitors the interface trigger event. If it is determined that the third-party application interface display area is triggered, it goes to step (3a); if it is determined that the setting area is triggered, it goes to step (3b);

[0112] (3a) Combination Figure 12 As shown, the main control application forwards the corresponding trigger event to the third-party application, and the third-party application responds to the trigger event; the third-party application updates its screen texture information on the virtual display screen, and the main control application refreshes the third-party application interface display area according to the updated screen texture information.

[0113] If the trigger area corresponds to the area corresponding to the third-party application app1, the main control application will perform coordinate conversion on the trigger event and distribute the corresponding trigger event to virtual display screen 1. Virtual display screen 1 will feedback the trigger event to app1, and app1 will respond to the trigger event. If the trigger area corresponds to the area corresponding to the third-party application app2, the main control application will perform coordinate conversion on the trigger event and distribute the corresponding trigger event to virtual display screen 2. Virtual display screen 2 will feedback the trigger event to app2, and app2 will respond to the trigger event.

[0114] (3b) The main control application determines whether the current setting area displays the live setting layout. If the scene add key is triggered, it proceeds to step (4a); if a scene thumbnail is triggered, it proceeds to step (4b); if a setting key is triggered, it proceeds to step (4c);

[0115] (4a) The settings area of ​​the master application is displayed as follows Figure 14 The scene creation layout shown in the figure includes selection keys such as HDMI1, HDMI2, USB, and CAM, as well as a picture loading key. These selection keys correspond to the image acquisition interface on the live broadcast machine. Clicking the picture loading key can load picture resources in the system library or in the external storage card. In short, the specific implementation can be set according to specific needs, so that the user can select all or part of the image data that the live broadcast machine can obtain here; after the selection is completed, go to step (5a);

[0116] (4b) The master application shares the image data corresponding to the scene with the third-party application, and the third-party application interface switches to the interface corresponding to the scene. The third-party application interface display area of ​​the master application is synchronously updated to display the third-party application interface. The specific implementation steps are as follows:

[0117] 1) The master application obtains the image data information corresponding to the scene thumbnail and processes the image data information according to the image editing settings selected when the corresponding scene was created;

[0118] 2) The main control application shares the processed image data information with the third-party application, and the third-party application draws the processed image data information on its corresponding virtual display screen;

[0119] 3) The main control application extracts the image texture information on the virtual display screen and draws it in the display area of ​​the third-party application interface.

[0120] (4c) If the main control application determines that the music setting key is triggered, it proceeds to step (5b); if it determines that the volume setting key is triggered, it proceeds to step (5c);

[0121] (5a) The settings area of ​​the master application is displayed as follows Figure 15 The scene editing layout shown in the figure includes a scene editing effect area, a layer thumbnail area, and an image editing button area. The layer thumbnail area displays the image data information input by the image access resource selected in step (4a). Drag the layer in the layer thumbnail to enter the scene editing effect area. Combined with the image editing button area, each layer can be subjected to image editing operations such as beautification, adding / removing borders, and changing backgrounds. The scene editing effect area displays the image editing effect. After the scene editing is completed, the setting area of ​​the main control application returns to the live broadcast setting layout. At this time, the new scene is added to the existing scene thumbnail.

[0122] (5b) The settings area of ​​the master application is displayed as follows Figure 16 In the music loading layout shown, if the add key is triggered, the audio resources in the system and the external storage card are loaded; after loading is complete, the main control application can perform operations such as play and pause control on the music.

[0123] (5c) The settings area of ​​the master application is displayed as follows Figure 17 The volume setting layout shown in the figure includes an input volume setting layout and an output volume setting layout, which can adjust the audio input volume and audio output volume respectively. In the present invention, the audio output volume can control all audio output interfaces on the device. The application layer opens to users some interface controls related to application functions, such as monitoring corresponding to the audio playback interface (Speaker), headphone (Headphone), and Bluetooth headset output volume; dragging the volume bar corresponding to live broadcast and recording can adjust the synthesized volume. This sound can only be heard by live viewers or playing recorded audio and video files. The corresponding audio data comes from the audio input interface; the audio input volume corresponds to all audio input interfaces, such as recording interface Mic, USB, HDMI-IN ( Figure 17 The main control application and third-party applications share audio data based on the input volume and output volume settings. When the volume bar is dragged to the bottom, it indicates muted, that is, the main control application does not share the data transmitted by the audio input interface with third-party applications.

[0124] As an embodiment, the live broadcast interface interaction of the live broadcast machine in the present invention can also include interactive scenarios in which multiple third-party applications are running simultaneously, such as Figure 13 As shown, a trigger pop-up window can be set in the upper right corner of the setting area to facilitate users to switch to multi-platform live broadcast scenes, or to use left, right, diagonal sliding triggers or other gesture triggers.

[0125] like Figure 18 、 Figure 19 As shown, in multicast mode, the display interface layout of the main control application has not changed significantly, except that the third-party application interface display area is increased. For example, when no new application is opened, the default display is as follows Figure 18 The main control application home page interface shown in the figure, and after opening the new application app2, the interface is displayed as follows Figure 19 The interface of the newly added application app2 is shown; the main control application can determine the third-party application selected by judging the user's trigger information.

[0126] In summary, the present invention modifies the existing Android system framework. Without modifying third-party applications, it changes the scheduling path of third-party applications for device hardware interfaces. The master application now manages and schedules hardware interfaces such as electronic display screens, audio input interfaces, audio output interfaces, and image acquisition interfaces. Third-party applications must schedule these hardware interfaces through the master application, thus avoiding resource conflicts caused by multiple third-party applications calling hardware interfaces simultaneously. The present invention displays the third-party application interface through the master application, facilitating the master application's operational control of the third-party application, especially the sharing of audio and image resources, and enabling the master application to effectively control the audio and image resources and interface display of the third-party application.

[0127] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. A third-party application interface display method is applied to a live broadcast machine, wherein the live broadcast machine adopts the Android system, has a main control application and can run third-party applications, and is provided with an electronic display screen, characterized in that: The electronic display screen only displays the interface of the main control application, and the interface of the main control application includes a third-party application interface display area. The method includes the following steps: (1) The master control application creates a virtual display screen according to preset specific parameters, wherein the preset specific parameters include an ID identifier and pixel information corresponding to the virtual display screen; the master control application creates a virtual display screen according to the preset specific parameters, which is based on the Android system architecture design and uses the createVirtualDisplay function in DisplayManager according to business needs to create a virtual display screen VirtualDisplay with preset specific parameters; (2) When a third-party application is triggered, the master application specifies a virtual display screen for it, and the third-party application draws the screen texture information of its interface on the corresponding virtual display screen according to the ID identifier corresponding to the virtual display screen, and the master application obtains the screen texture information in the corresponding virtual display screen through the ID identifier; The master application specifies a virtual display screen for the third-party application by specifying the activity start of the third-party application in the ID identifier corresponding to the corresponding virtual display screen; (3) the main control application performs image processing on the screen texture information of the third-party application and then draws the image in the third-party application interface display area; (4) The main control application monitors the interface triggering event and updates the third-party application interface display area according to the triggering event.

2. The third-party application interface display method according to claim 1, characterized in that: In step (4), the master control application monitors the interface trigger event and updates the third-party application interface display area according to the trigger event, including the following steps: If it is determined that the third-party application interface display area is triggered, the main control application forwards the corresponding trigger event to the third-party application through the virtual display screen, and the third-party application responds to the trigger event and updates its screen texture information on the virtual display screen; The main control application updates the third-party application interface display area according to the updated screen texture information.

3. The third-party application interface display method according to claim 1, characterized in that: The live broadcast machine is also provided with an image acquisition interface; the interface of the main control application also includes a setting area. In the initial state, the setting area displays the live broadcast setting layout, and the live broadcast setting layout includes scene thumbnails and a scene add button.

4. The third-party application interface display method according to claim 3, characterized in that: The master control application listens for interface trigger events. If it is determined that the scene add key is triggered, the setting area switches to the scene creation layout. The scene creation layout includes a selection key for the image acquisition interface. After the master control application detects that the interface selection is completed and the confirmation event is triggered, the setting area switches to the scene editing layout.

5. The third-party application interface display method according to claim 4, characterized in that: The scene editing layout includes a scene editing effect area, a layer thumbnail area and an image editing button area. The layer thumbnail area displays the image data information input by the selected image acquisition interface respectively; the layers in the layer thumbnail area can be dragged to the scene editing effect area, and the image editing button area includes image editing operation keys, and the image editing operation keys can perform image editing operations on the layers in the scene editing effect area; after the scene editing is completed, the setting area returns to the live broadcast setting layout, and the edited scene is added to the scene thumbnail.

6. The third-party application interface display method according to claim 5, characterized in that: In step (4), the master control application monitors the interface trigger event and updates the third-party application interface display area according to the trigger event, including the following steps: If it is determined that the scene thumbnail is triggered, the main control application obtains the image data information corresponding to the scene thumbnail and processes the image data information according to the image editing settings selected when the corresponding scene is created; The main control application shares the processed image data information with the third-party application, and the third-party application draws the processed image data information on its corresponding virtual display screen; The main control application extracts the image texture information on the virtual display screen and draws it in the third-party application interface display area.

7. The third-party application interface display method according to claim 6, characterized in that: The main control application shares the processed image data information with the third-party application, including the following steps: Create a virtual camera hardware abstraction layer based on the Android system architecture, and establish an access path for the third-party application to access the virtual camera hardware abstraction layer; The master control application transmits the processed image data information to the virtual camera hardware abstraction layer through an inter-process communication strategy; The application layer of the third-party application obtains the image data information through the virtual camera hardware abstraction layer.

8. The third-party application interface display method according to claim 3, characterized in that: The live broadcast setting layout also includes a volume setting key; when the main control application detects that the volume setting key is triggered, the setting area switches to the volume setting layout, which includes an input volume setting layout and an output volume setting layout.

9. The third-party application interface display method according to claim 8, characterized in that: The main control application and the third-party application share audio data according to the settings of input volume and output volume.

10. The third-party application interface display method according to any one of claims 1 to 9, characterized in that: The third-party application interface display area can display multiple third-party applications. The main control application creates multiple virtual display screens and specifies different virtual display screens for different third-party applications.

Images