Data sharing method, electronic equipment and system

Abstract

The invention relates to the technical field of communication, and discloses a data sharing method, electronic equipment and a system. The method comprises the following steps: receiving first streaming media data from a sending end; decoding the first streaming media data to obtain second streaming media data in a preset format; transmitting the second streaming media data to a virtual drive program in a core layer of the native system; the virtual drive program is identified as a virtual audio and video node by the conference application; and after directly reading the second streaming media data of the virtual audio and video node through the conference application, displaying the second streaming media data and sharing the second streaming media data to the remote equipment. According to the invention, instant sharing of the screen transmission picture from the sending end is realized, private interface development does not need to be carried out for each conference application, compatibility and maintainability are good, and when desktop sharing is carried out on the remote conference, once streaming media data sent by the sending end are received, the remote conference can be shared in real time. And the streaming media data can be directly shared to the remote equipment of the teleconference.

Description

Technical Field

[0001] This invention relates to the field of data transmission, and in particular to a data sharing method, electronic device, and system. Background Technology

[0002] With the development of communication technology, video conferencing is becoming increasingly common. Video conferencing refers to a network meeting method in which two or more individuals or groups in different locations exchange audio, video and file data through transmission lines and multimedia equipment to achieve data sharing, thereby enabling real-time and interactive communication.

[0003] In existing technologies, video conferencing is mainly achieved by installing conferencing applications on different devices. These applications typically only enable the sharing of camera or desktop data on the device itself, but there are still problems with sharing screen feeds received from remote devices.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] Firstly, this application provides a data sharing method for sharing a screen display from a remote device to a remote device, thereby achieving a screen sharing function. The data sharing method is applied to an electronic device and includes:

[0006] Establish a remote meeting with at least one remote device using a conferencing application;

[0007] Receive the user's first action;

[0008] In response to the first action, the local screen is shared to the remote device via the conferencing application, specifically:

[0009] Receive the first streaming media data from the sender;

[0010] The first streaming media data is decoded to obtain a second streaming media data in a preset format; wherein the preset format is a format supported by the core layer of the native system of the electronic device.

[0011] The second streaming media data is transmitted to a virtual driver set in the core layer of the native system; the virtual driver is recognized by the conferencing application as a virtual audio and video node;

[0012] After directly reading the second streaming media data of the virtual audio and video node through the conference application, the second streaming media data is displayed and shared to the remote device.

[0013] In one possible implementation, the method also includes:

[0014] Detecting the connection of audio and video devices;

[0015] When an audio / video device is detected to be connected, the conferencing application blocks the audio / video data sent by the device until the electronic device disconnects from the sending end or the conferencing application stops sharing with the remote device.

[0016] In one possible implementation, the method also includes:

[0017] Receive the user's second action;

[0018] According to the second operation, annotation handwriting is superimposed on the display window of the second streaming media data;

[0019] The annotation handwriting is shared to the remote device.

[0020] In one possible implementation, the sending end is a screen sharing device, and the first streaming media data is desktop data sent from a personal terminal connected to the screen sharing device to the screen sharing device.

[0021] Alternatively, the sending end is a screen sharing application installed on a personal terminal, and the first streaming media data is the desktop data of the personal terminal captured by the screen sharing application.

[0022] In one possible implementation, transmitting the second streaming media data to a virtual driver configured in the core layer of the native system specifically includes:

[0023] The second streaming media data is sent to the HAL layer through a preset interface, and then the HAL layer sends the second streaming media data to the virtual driver of the core layer.

[0024] The virtual driver includes a virtual camera driver and a virtual microphone driver, and the virtual audio / video node includes a virtual camera node and a virtual microphone node.

[0025] In one possible implementation, the method also includes:

[0026] The presence of a virtual camera driver is monitored via a hot-swappable thread; the hot-swappable thread is used to monitor whether a virtual camera driver is connected.

[0027] When the hot-plug thread detects the access of the virtual camera driver, the corresponding virtual camera node is loaded into the camera hardware abstraction layer so that the conferencing application can identify the virtual camera node.

[0028] In one possible implementation, the method also includes:

[0029] The hot-swappable thread monitors whether the connected virtual camera driver has been closed.

[0030] When the hot-plug thread detects that the virtual camera driver is closed, the virtual camera node that has been loaded into the camera hardware abstraction layer is removed.

[0031] In one possible implementation, the format of the first streaming media data is one or more of rtsp, rtmp, rtp, or srtp;

[0032] The preset format is one or more of the following: YUV format, RGB format, JPEG format, or PCM format.

[0033] In a second aspect, the present invention provides an electronic device, comprising:

[0034] At least one processor; and,

[0035] A memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the data sharing method described in the first aspect.

[0037] To address the aforementioned technical problems, in a third aspect, the present invention provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the data sharing method described in the first aspect.

[0038] To address the aforementioned technical problems, in a fourth aspect, the present invention provides a data sharing system, including a transmitting end and an electronic device;

[0039] The sending end is used to connect to the terminal device, and after obtaining the first streaming media data from the terminal device, sends the first streaming media data to the electronic device.

[0040] The electronic device is used to establish a remote conference with a remote device via a conferencing application and to perform the data sharing method described in the first aspect.

[0041] Compared with related technologies, the beneficial effects of this invention are as follows: In this invention, an electronic device establishes a remote conference with at least one remote device through a conferencing application. The electronic device can not only share its local screen to the remote device through the conferencing application, but also directly share streaming media data from the sending end to the remote device, achieving real-time sharing of the screen from the sending end. The electronic device transmits the format-converted streaming media data to a virtual driver program set in the core layer of the native system. The virtual driver program is recognized as a virtual audio / video node by the conferencing application. The electronic device directly reads the virtual audio / video node through the conferencing application to obtain streaming media data, thereby displaying the streaming media data locally and sharing it to the remote device. Since the virtual audio / video node can be accessed by the conferencing application, the conferencing application can directly read the data from the virtual audio / video node for display and directly share the received streaming media data to the remote device. This method does not require the development of private interfaces for each conferencing application, thus offering good compatibility and maintainability, high portability, and eliminating the need for complex user operations. This achieves the effect of directly sharing streaming media data to the remote device in a remote conference once the streaming media data sent by the sending end is received during desktop sharing. Attached Figure Description

[0042] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise stated, the figures in the drawings do not constitute a limitation on scale.

[0043] Figure 1 This is a flowchart illustrating the first data sharing method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating an application scenario of a data sharing method provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the interactive interface of the first data sharing method provided in the embodiments of the present invention;

[0046] Figure 4 This is a flowchart illustrating the second data sharing method provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the first data sharing method provided in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the second data sharing method provided in an embodiment of the present invention;

[0049] Figure 7 This is a flowchart illustrating the third data sharing method provided in this embodiment of the invention;

[0050] Figure 8 This is a schematic diagram of the interactive interface of the second data sharing method provided in the embodiments of the present invention;

[0051] Figure 9 This is a schematic diagram of the third data sharing method provided in the embodiments of the present invention;

[0052] Figure 10 This is a schematic diagram of the fourth data sharing method provided in the embodiments of the present invention;

[0053] Figure 11 This is a schematic diagram of the fifth data sharing method provided in the embodiments of the present invention;

[0054] Figure 12 This is a flowchart illustrating the fourth data sharing method provided in this embodiment of the invention;

[0055] Figure 13 This is a flowchart illustrating the fifth data sharing method provided in this embodiment of the invention;

[0056] Figure 14 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of the present invention;

[0057] Figure 15 This is a schematic diagram of the architecture of another electronic device provided in an embodiment of the present invention;

[0058] Figure 16 This is a schematic diagram of the architecture of a data sharing system provided in an embodiment of the present invention. Detailed Implementation

[0059] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0061] It should be noted that, unless otherwise specified, the various features in the embodiments of the present invention can be combined with each other, and all are within the protection scope of the present invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed differently from the module division in the device or the order shown in the flowchart.

[0062] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

[0063] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0064] With the development of communication technology, data sharing has become increasingly widespread. Among them, screen sharing refers to a data sharing method that transmits data collected by one device to another device for playback. It is a commonly used data interaction method in the field. Furthermore, with the advancement of screen sharing technology, its application scenarios are becoming more and more diversified, such as in meetings, education, and other scenarios.

[0065] In some scenarios, users may also need to combine screen sharing with conferencing applications, such as sharing screen footage within a conferencing application. In electronic conferencing scenarios, after an electronic device establishes a remote meeting and shares its screen with a remote device, if it receives a screen-sharing image from the sender, the user needs to click "share" to share the image with the remote device. To achieve the function of directly sharing the received screen-sharing image to the remote device, one approach is to jointly develop the conferencing application and the electronic device, creating a private Application Programming Interface (API) between them. The electronic device uses this API to transmit the received screen-sharing image to the conferencing application, which then shares it with the remote device. However, this approach requires both the electronic device and the conferencing application to maintain a unified set of private interfaces. When one party modifies the interface for later management and maintenance, the other party also needs to be modified, resulting in poor maintainability. Furthermore, because it requires development support from the conferencing application side, it cannot be directly compatible with all conferencing applications on the market, resulting in poor compatibility, hindering the formation of a robust ecosystem, and causing low portability efficiency.

[0066] To address this issue, this application embodiment decodes the received streaming media data and converts it into a format supported by the core layer of the electronic device's native system. The decoded streaming media data is then sent to a virtual driver program set up in the core layer of the native system. This virtual driver program can be directly recognized as a virtual audio / video node by the conferencing application, allowing the application to directly read and display the data from the virtual audio / video node. This data can also be shared with the other side of the remote conference. This eliminates the need for complex user operations and the development of proprietary interfaces for each conferencing application. By directly using the system-level virtual driver program, the effect of sharing streaming media data directly to the remote device in a remote conference upon receiving the data from the sending end during desktop sharing is achieved.

[0067] One embodiment of this application provides a data sharing method applied to electronic devices, such as... Figure 1 As shown, it includes:

[0068] In step 201, a remote conference is established with at least one remote device via a conferencing application.

[0069] The remote conference refers to a meeting method that shares images, videos, audio, and other data between one or more electronic devices to achieve real-time communication in a remote environment. The conference application is an application used to implement the remote conference. The conference application can be an application pre-installed on the electronic device or a third-party application installed by the user. This embodiment does not limit this. The remote device is a device other than the electronic device. After installing the conference application on both the electronic device and the remote device, and performing the operation of starting or joining a conference, a remote conference is established. This embodiment will further elaborate on the method after providing a complete explanation, combined with... Figure 2 The actual application scenarios for conferencing applications, remote devices, and electronic devices will be described in detail here, and will not be repeated here.

[0070] In step 202, a first user operation is received; the first operation is the user's operation to enable screen sharing. In specific application scenarios, the first operation may be clicking the corresponding function button to enable screen sharing on the meeting application, such as... Figure 3 The "Share Screen" button is shown.

[0071] In step 203, in response to the first operation, the local screen is shared to the remote device via the conferencing application.

[0072] In step 204, the first streaming media data is received from the sending end.

[0073] In practical use, the sending end is a screen sharing device, and the first streaming media data is desktop data sent from a personal terminal connected to the screen sharing device to the screen sharing device; alternatively, the sending end is a screen sharing application installed on a personal terminal, and the first streaming media data is desktop data captured by the screen sharing application from the personal terminal, to support different types of screen sharing. The screen sharing device and the personal terminal can be connected via a Type-C interface.

[0074] In step 205, the first streaming media data is decoded to obtain second streaming media data in a preset format; wherein, the preset format is a format supported by the core layer of the native system of the electronic device; in an optional embodiment, the native system is the Android system, the core layer is the kernel layer of the Android system, and the format of the first streaming media data is one or more of rtsp, rtmp, rtp, or srtp; the preset format is one or more of YUV, RGB, JPEG, or PCM formats to support streaming media data in multiple formats. In some embodiments, the second streaming media data includes audio data and video data, wherein the audio data is in PCM format, and the video data is one or more of YUV, RGB, or JPEG formats.

[0075] In step 206, the second streaming media data is transmitted to a virtual driver set in the core layer of the native system; the virtual driver is recognized by the conferencing application as a virtual audio and video node.

[0076] The virtual driver includes a virtual camera driver and a virtual microphone driver, and the virtual audio / video node includes a virtual camera node and a virtual microphone node.

[0077] In step 207, after directly reading the second streaming media data of the virtual audio and video node through the conference application, the second streaming media data is displayed and shared to the remote device.

[0078] The process of directly reading the second streaming media data from the virtual audio and video node through the conferencing application can be as follows: the conferencing application directly calls the native system's built-in interface to read the second streaming media data from the virtual audio and video node. The number of remote devices can be one or more. It should be noted that sharing the second streaming media data to the remote device can be done by directly sending the second streaming media data to the remote device, or by converting the format of the second streaming media data before sending it to the remote device. This invention does not limit the specific implementation of the conferencing application sharing the second streaming media data to the remote device.

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the application scenarios of this embodiment are described below in conjunction with the accompanying drawings, such as... Figure 2 As shown, the personal terminal generates first streaming media data based on the local screen and sends it to an electronic device. In practical use, this can be sent to a screen sharing application on the electronic device. The screen sharing application decodes the first streaming media data to obtain second streaming media data and transmits it to a virtual audio / video node. The conferencing application reads the second streaming media data from the virtual audio / video node and shares it with the remote device, thus achieving screen sharing through the conferencing application. For ease of distinction, this embodiment refers to the screen sharing application installed on the personal terminal as the screen sharing sender and the screen sharing application installed on the electronic device as the screen sharing receiver.

[0080] The personal terminal, electronic device, and remote device can be any device with display or projection capabilities, such as a tablet, laptop, mobile phone, television, display screen, projector, or augmented reality (AR) device. The personal terminal and electronic device can be the same type of device (e.g., both are mobile phones) or different types of devices (e.g., the personal terminal is a mobile phone and the electronic device is a computer).

[0081] Data transmission between the personal terminal and the electronic device can be accomplished using any transmission technology, such as direct connection transmission via cable, network transmission, Bluetooth transmission, direct connection transmission via Wireless Local Area Network (WLAN), etc. This invention does not limit the specific technology used.

[0082] Furthermore, the personal terminal can rely on its own capabilities to transmit data. For example, a screen sharing application can be installed on both the personal terminal and the electronic device. The screen sharing application uses the personal terminal's own wireless communication capabilities to send the first streaming media data to the electronic device. In this case, the sending end is the screen sharing application.

[0083] The personal terminal can also rely on other devices connected to it for data transmission. For example, if the personal terminal is connected to a screen sharing device, the personal terminal will transmit the first streaming media data to the screen sharing device through a direct connection. The screen sharing device will then send the first streaming media data to the electronic device through communication methods such as Bluetooth or wireless network. In this case, the sending end is the screen sharing device.

[0084] Data transmission between the personal terminal and the electronic device can be implemented based on various network transmission protocols, such as Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), H.323 protocol, Session Initialization Protocol (SIP), Web Real-Time Communications (WebRTC), and custom transmission protocols. This invention does not limit the specific network transmission protocols used.

[0085] In this embodiment, the electronic device establishes a remote conference with at least one remote device through a conferencing application. The electronic device can not only share its local screen to the remote device through the conferencing application, but also share streaming media data from the sending end to the remote device, thus achieving screen sharing from the sending end. The electronic device transmits the format-converted streaming media data to a virtual driver set in the core layer of the native system. The virtual driver is recognized as a virtual audio / video node by the conferencing application. The electronic device directly reads the virtual audio / video node through the conferencing application to obtain the streaming media data, thereby displaying the streaming media data locally and sharing the streaming media data to the remote device. Because the virtual audio / video node can be accessed by the conferencing application without setting up additional private interfaces, it has good compatibility, maintainability, and portability.

[0086] In practical use, to avoid the actual audio and video devices affecting the screen sharing operation, this embodiment also provides a preferred implementation method, namely, the method further includes:

[0087] The system detects the access of audio and video devices. Upon detection of such access, the conferencing application blocks audio and video data transmitted by the device until the electronic device disconnects from the sending end or the conferencing application stops sharing with the remote device. In other words, during screen sharing, when a new audio and video device is detected, the sharing of screen data is maintained, and the system does not switch to the new device, thereby preventing the new device from interrupting the screen sharing function. The audio and video devices include cameras and microphones.

[0088] In a preferred embodiment, such as Figure 4 As shown, the method also includes:

[0089] In step 301, the user's second operation is received.

[0090] In step 302, according to the second operation, annotations are overlaid on the display window of the second streaming media data; in practical application scenarios, the display window is the window of the conferencing application used to display the first streaming media data. After reading the second streaming media data from the virtual audio and video node, the conferencing application displays the second streaming media data on the display window, such as... Figure 8 As shown; the second operation can be to perform a sliding motion on the display window after opening the annotation function, such as in Figure 8 Click the interactive annotation button on the interface of the meeting application shown to open the annotation function, and then make a sliding mark on the display window; wherein, the sliding action can be implemented by the mouse or by the touch function, and this embodiment does not limit the specific implementation method of the action.

[0091] In step 303, the annotation handwriting is shared to the remote device. One optional implementation of sharing the annotation handwriting to the remote device is that the electronic device acquires the annotation handwriting through a conferencing application and then shares it with the remote device via the conferencing application. A second optional implementation is that the electronic device acquires the annotation handwriting through an annotation application, transmits the annotation handwriting to the conferencing application, and then shares the annotation handwriting with the remote device via the conferencing application. The annotation application is an application installed on the electronic device for implementing annotation functions. During transmission, the annotation handwriting can be represented by the coordinates of the handwriting points on the first streaming media data and the writing time of the handwriting points. The annotation handwriting and the second streaming media data are sent separately to the remote device. In the second optional implementation described above, the annotation handwriting can be transmitted to the conference application through the interface between the conference application and the remote device. Then, the conference application shares the annotation handwriting and the second streaming media data with the remote device through different transmission channels. The remote device combines the annotation handwriting and the second streaming media data to display them together, thereby supporting the annotation function of the screen sharing and realizing the synchronous sharing of annotation handwriting and screen sharing.

[0092] In practical use, transmitting the second streaming media data to the virtual driver set in the core layer of the native system specifically includes:

[0093] The second streaming media data is sent to the HAL layer (also called the Hardware Abstraction Layer) via a preset interface, and then the HAL layer sends the second streaming media data to the virtual driver in the core layer. The preset interface is a proprietary interface pre-developed and provided by those skilled in the art, and its main function is to enable the interface between the application layer and the core layer through the HAL layer. Before the HAL layer sends the second streaming media data to the virtual driver in the core layer, the method further includes: the electronic device opening the virtual driver, specifically opening the virtual camera driver and the virtual microphone driver.

[0094] This embodiment uses the V4L2 video device driver framework to implement a virtual camera driver and the snd-aloop virtual sound card module to implement a virtual microphone driver as an example.

[0095] V4L2, short for Video for Linux 2, is a kernel driver for video devices in Linux. It's a set of general API interfaces for capturing image, video, and audio data under the Linux operating system. With appropriate video capture devices and corresponding drivers, it can capture images, videos, and audio. V4L2 supports three methods for image capture: memory mapping (mmap), direct reading (read), and user pointers. This embodiment mainly uses memory mapping as an example. v4l2loopback is a module in the V4L2 framework that allows the creation of virtual video devices and the passing of input streams to other applications. This enables the simulation of a virtual camera even without a hardware camera. Its core is the virtual camera driver v4l2lookback.ko, located in the system's kernel layer. Figure 5 As shown.

[0096] The v4l2lookback.ko file is recognized by the conferencing application as a virtual camera node. This virtual camera node is typically named " / dev / videoX", meaning it's a videoX file located in the " / dev / " directory, where X is an integer. Which virtual camera node corresponds to the virtual camera driver is determined by those skilled in the art based on their needs. In actual use, loading v4l2lookback.ko generates the corresponding virtual camera node, which can also be understood as generating the device node file corresponding to the virtual camera driver. v4l2lookback provides an external interface at the HAL layer for writing video data into v4l2lookback.ko, i.e., writing the video data to the corresponding device node file.

[0097] When implementing a virtual microphone driver using the snd-aloop module for a virtual sound card, the virtual microphone driver is virtualaudio.ko. In practice, when virtualaudio.ko is loaded, corresponding write nodes and read nodes (collectively referred to as virtual microphone nodes) are generated, such as... Figure 6 As shown, the write node is card 0device0, and the read node is card 0device1. The snd-aloop module also provides an external pcm_write() interface, which is used to write audio data into virtualaudio.ko (specifically, to the write node).

[0098] The preset interface is obtained by encapsulating the external interface provided by v4l2lookback and the pcm_write() interface provided by the snd-aloop module into a custom Java Native Interface (JNI) for the API layer.

[0099] The screen sharing application located at the application layer calls the preset interface, enabling video and audio data to travel from the application layer through the API layer to the HAL layer. Video data is then written to v4l2lookback.ko via the external interface provided by v4l2lookback in the HAL layer, and audio data is written to virtualaudio.ko via the pcm_write() interface. In practical use, the external interface provided by v4l2lookback and the pcm_write() interface can also be encapsulated separately to obtain preset interfaces for video data and audio data, respectively.

[0100] In practical applications, after setting up a virtual camera driver at the core layer, the conferencing application at the application layer cannot directly detect it. To solve this problem, such as... Figure 7 As shown, the method also includes:

[0101] In step 401, a hot-plug thread is pre-established in the camera hardware abstraction layer.

[0102] In step 402, a hot-swappable thread is used to monitor whether a virtual camera driver is connected; the hot-swappable thread is used to monitor whether a virtual camera driver is connected.

[0103] In step 403, when the hot-plug thread detects the virtual camera driver access, the corresponding virtual camera node is loaded into the camera hardware abstraction layer so that the conferencing application can identify the virtual camera node. The camera hardware abstraction layer can be understood as... Figure 5 The camera HAL layer in the system. When the hot-plug thread detects the presence of a virtual camera node, it calls the deviceAdded() interface, which executes...

[0104] The `deviceAdded(" / dev / videoX")` method adds the virtual camera node ` / dev / videoX` to the camerahal layer, enabling the conferencing application at the application layer to recognize the virtual camera driver and automatically detect its access. This allows the conferencing application to directly read video data from the virtual camera node. Notably, due to the different architectures of the virtual microphone driver and the virtual camera driver, the conferencing application can directly identify the virtual microphone node without additional processing via a hot-plugging thread.

[0105] Correspondingly, the method also includes: monitoring whether the connected virtual camera driver has been closed via the hot-plug thread; when the hot-plug thread detects that the virtual camera driver has been closed, removing the virtual camera node that has been loaded into the camera hardware abstraction layer. This is achieved by the hot-plug thread calling the deviceRemoved() interface.

[0106] The `deviceRemoved(" / dev / videoX")` method removes the virtual camera node from the camera HAL layer, thereby disabling the conferencing application's awareness of the virtual camera driver and stopping video data sharing. This allows video data sharing to be directly shut down when the first stream of streaming data sent by the sending end terminates.

[0107] In specific application scenarios, this embodiment also provides an exit screen sharing function, such as providing an exit screen sharing button in a meeting application, such as... Figure 8 The "Stop Sharing" button shown in the image generates an exit sharing message when the user clicks it. This message is then broadcast to the sharing application, which in turn closes the virtual camera driver upon receiving the exit sharing message.

[0108] The exit screen sharing message can also originate from the sending end. Specifically, when the user presses the corresponding exit screen sharing button on the screen sharing device connected to their personal terminal, the screen sharing device sends the exit screen sharing message to the electronic device.

[0109] After providing a data sharing method in the first embodiment, the second embodiment of this application will further provide a data sharing method in order to elaborate on the above method from the perspective of the sending end and to further analyze its design principles.

[0110] This embodiment provides a data sharing method, including:

[0111] The first streaming media data is sent to the electronic device so that the electronic device can execute the data sharing method described in the first embodiment. In this embodiment, the executing entity of the data sharing method is the sending end.

[0112] Specifically, when the sending end is a screen sharing application installed on a personal terminal, the first streaming media data is obtained by the screen sharing application capturing desktop data from the personal terminal; when the sending end is a screen sharing device connected to the personal terminal, the first streaming media data is desktop data received from the personal terminal.

[0113] This embodiment, when used in conjunction with the first embodiment, enables the conferencing application to directly obtain data from virtual audio and video nodes, thereby eliminating the need to provide additional interfaces for the conferencing application. This achieves compatibility with various third-party conferencing applications, improves portability, and facilitates later maintenance.

[0114] After providing a data sharing method described in the first embodiment and a data sharing method described in the second embodiment, the third embodiment of this application will further provide a data sharing method in order to elaborate on the method in the first embodiment from the perspective of conference application and to further analyze its design principles.

[0115] This embodiment provides a data sharing method, including:

[0116] The second streaming media data is obtained from the virtual audio and video node and shared with the remote device; wherein the second streaming media data is transmitted to the virtual audio and video node by the screen sharing application.

[0117] The acquisition of the second streaming media data from the virtual audio and video node can be achieved using a custom interface of the screen sharing application.

[0118] In another embodiment, the acquisition of the second streaming media data from the virtual audio and video node can also be implemented using the system's built-in interface, that is, the acquisition of the second streaming media data from the virtual audio and video node specifically includes:

[0119] The native system's built-in onPreviewFrame() interface is used to obtain video data from the virtual camera node, and the native system's built-in pcm_read() interface is used to obtain audio data from the virtual microphone node. In other words, the native system's camera architecture is used to obtain video data, and the native system's MediaRecord process is used to obtain audio data.

[0120] This embodiment, when used in conjunction with the first and second embodiments, can directly obtain data from virtual audio and video nodes without the need for additional data acquisition interfaces, offering good compatibility and facilitating development and subsequent management.

[0121] This application also elaborates on the implementation process of the features of the present invention in specific application scenarios based on the methods described in the first, second and third embodiments, and by using technical descriptions in related scenarios.

[0122] This embodiment uses Figure 2 The desktop of the personal terminal shown is projected to a screen sharing application on an electronic device, and then the screen sharing screen is shared to a remote device using a conferencing application on the electronic device. For example, the system used by the electronic device is the Android system, and the conferencing application on the electronic device is a third-party application, also called a third-party conferencing Android application package (apk). The data sharing method described in this embodiment includes: setting up a virtual camera driver and a virtual microphone driver in the electronic device in advance.

[0123] like Figure 9 As shown, the virtual camera driver can receive data from the camera, screen sharing device, or RTSP stream. Since the virtual camera driver uses the native camera architecture of Android, third-party applications can be perfectly compatible with the screen sharing architecture of the virtual camera driver. That is, third-party applications can directly connect to the native Android interface, saving the need to connect to the API interface of the screen sharing application, which reduces the workload of development and greatly enhances the compatibility of the app.

[0124] like Figure 10 As shown, the virtual microphone driver has a similar architecture to the virtual camera driver. It can receive audio data from a microphone, network audio stream, or screen sharing device. The virtual microphone driver can follow the native Android MediaRecord process, so third-party APKs can be compatible with the screen sharing architecture of the virtual microphone driver.

[0125] The specific implementation of the screen sharing process is as follows: Figure 11As shown, the personal terminal sends the first streaming media data to a screen sharing application in the electronic device. The screen sharing application receives the first streaming media data and displays it on the screen sharing application window, thus realizing the screen sharing function. Furthermore, the screen sharing application decodes the first streaming media data into second streaming media data (including video and audio data) in a format supported by the Android kernel layer. This second streaming media data is then transmitted to virtual drivers in the kernel layer, namely, a virtual camera driver and a virtual microphone driver. The conferencing application identifies the virtual camera driver as a virtual camera node and directly acquires video data from it. Similarly, it identifies the virtual microphone driver as a virtual microphone node and directly acquires audio data from it, thus obtaining complete second streaming media data. The conferencing application then shares this second streaming media data with a remote device, realizing the shared screen sharing function. The video data transmission process is as follows: Figure 5 and Figure 12 As shown, it specifically includes:

[0126] In step 501, the screen sharing receiver (i.e., the screen sharing application on the electronic device) receives video bitstream data (i.e., the video-related part of the first streaming media data) from screen sharing device projection, soft terminal projection, or network, and then decodes the YUV format video data (also known as YUV data).

[0127] In step 502, the interface is connected to the v4l2lookback hal layer via the JNI private interface, and then the YUV data is transmitted to the v4l2lookback.ko virtual driver layer.

[0128] In step 503, the third-party meeting APK obtains the YUV data from step 502 by calling the native Android camera architecture process, thereby realizing the function of sharing video, that is, transmitting the screen sharing screen to the third-party meeting APK so that the third-party meeting APK can share the screen sharing screen with remote devices.

[0129] The process of transmitting audio data is as follows Figure 6 and Figure 13 As shown, it specifically includes:

[0130] In step 601, the screen sharing receiver receives audio stream data (i.e., the audio-related part of the first streaming media data) from screen sharing device projection, soft terminal projection, or network, and then decodes the PCM format audio data (also known as PCM data).

[0131] In step 602, the virtual microphone audiohal layer is connected via the JNI private interface, and then the PCM data is transmitted to the virtualaudio.ko virtual driver layer.

[0132] In step 603, the third-party meeting APK obtains the PCM data from step 602 by calling the native Android MediaRecord architecture process, thereby realizing the function of sharing audio, that is, transmitting the audio data corresponding to the screen sharing screen to the third-party meeting APK so that the third-party meeting APK can share the audio data with remote devices.

[0133] The following will illustrate this with specific implementation code:

[0134] First, configure the virtual camera driver and provide the following interfaces for upper-layer applications:

[0135] The virtual camera interface is initialized using the method: `int initVirtualCamera(int width, int height, const char *format)`. During initialization, a virtual camera node is generated for the virtual camera. In one optional implementation, such as... Figure 11 The image shows / dev / video99.

[0136] The interface for writing data to the virtual camera is: int writeFrame(unsigned char*buffer, int framesize); This interface is used to write video data into the virtual camera driver and can be understood as the default interface for video data.

[0137] Close the virtual camera interface: int closeVirtualCamera(void).

[0138] The interface for determining whether a virtual camera is open is: bool isOpened().

[0139] The interface for determining whether an I-frame needs to be resent is: bool isNeedIFrame().

[0140] Secondly, configure the virtual microphone driver and provide the following interfaces for upper-layer applications:

[0141] Open the virtual microphone device interface: int virAdev_open_output_stream(int sampleRate,int channel_mask,int formats); When the virtual microphone driver is opened, virtual microphone nodes are generated, including the write node / dev / snd / pcmC99p0 and the read node / dev / snd / pcmC99c0.

[0142] The interface for writing PCM audio data to the virtual microphone driver is: int virAdev_out_writeFrame(unsigned char*buffer, int framesize); this interface calls the pcm_write() interface to write PCM audio data to the write node corresponding to the virtual microphone driver, which is the default interface for audio data.

[0143] Close the virtual microphone device interface: intvirAdev_close_output_stream(void).

[0144] The interface for determining whether a virtual microphone is open is: bool virAdev_isOpened().

[0145] Next, a hot-plug thread, VirCamHotplugThread, is added to the camera HAL layer. VirCamHotplugThread is used to detect whether the virtual camera driver has been connected. If the virtual camera driver is detected to be connected, the virtual camera node is added to the camera HAL layer, i.e., mVirParent->deviceAdded(" / dev / video99"). The connection status of the virtual camera can be obtained through the system's built-in interface public void onCameraAvailable(String cameraId).

[0146] In practical use, to facilitate user operation, a one-click exit function for screen sharing is also provided. Specifically:

[0147] When you receive a message to exit screen sharing, exit screen sharing.

[0148] The exit screen sharing message can be generated by the screen sharing device connected to the electronic device or a third-party application and transmitted to the screen sharing receiver. This embodiment takes the exit screen sharing message being generated by a third-party application as an example. Figure 3 and Figure 8 As shown, third-party applications provide a button to enable screen sharing (i.e. Figure 3 The "Share Screen" button and the function button to exit screen sharing (i.e.) Figure 8 The third-party application shares its local screen when the user clicks the "Stop Sharing" button. During the local screen sharing process, if the first streaming media data is received, the screen sharing function is entered. When the user clicks the "Exit Screen Sharing" button, the third-party application generates the exit screen sharing message and broadcasts the exit screen sharing message.

[0149] Correspondingly, the screen sharing receiving end listens for the exit screen sharing message in real time.

[0150] When VirCamHotplugThread detects a virtual camera driver connection, it adds the virtual camera node to the camera HAL layer. After receiving the first streaming media data from the personal terminal, the screen mirroring receiver decodes it to obtain the second streaming media data (including video and audio data), opens the virtual camera driver, and calls the `int writeFrame(unsigned char* buffer, int framesize)` interface to write video data into the virtual camera driver. It also opens the virtual microphone driver and calls the `int virAdev_out_writeFrame(unsigned char* buffer, int framesize)` interface to write audio data into the virtual microphone driver.

[0151] Meanwhile, the third-party application calls the `public void onCameraUnavailable(String cameraId)` interface to determine if the virtual camera driver is connected. Once connected, it uses the `Camera.open()` interface to check if the virtual camera driver is open. If open, it sets a callback for the `onPreviewFrame()` interface using the system's built-in `setPreviewCallback()` interface. The `onPreviewFrame()` interface reads video data from the virtual camera node. It also calls the `pcm_write()` interface to read audio data from the virtual microphone driver's read node, thus obtaining both video and audio data. This data is then shared with the remote device to achieve screen sharing.

[0152] When the receiving end detects the exit screen sharing message, it calls the int closeVirtualCamera(void) interface to close the virtual camera driver, and VirCamHotplugThread executes...

[0153] `mVirParent->deviceRemoved(" / dev / video99")` removes the virtual camera node from the camerahal layer. Third-party applications, in response to the user's exit from screen sharing, cease reading data from the virtual camera and virtual microphone read nodes.

[0154] This embodiment also supports the annotation function, which allows users to add annotations to the first streaming media data on the screen receiving end window. The screen receiving end transmits the annotations to a third-party application through the interface, and the third-party application shares the annotations with a remote device.

[0155] The data sharing method described in this embodiment is compatible with most third-party meeting APKs on the market, such as Tencent Meeting and Lark. Furthermore, it boasts strong portability, reduces development workload, and supports one-click screen sharing, making it easy to operate.

[0156] Based on the foregoing embodiments, such as Figure 14 As shown, this embodiment also provides an electronic device, including: at least one processor 21; and a memory 22 communicatively connected to the at least one processor 21; wherein the memory 22 stores instructions executable by the at least one processor 21, the instructions being executed by the at least one processor 21 to enable the at least one processor 21 to perform the data sharing method described in the first embodiment above.

[0157] The processor 21 and the memory 22 can be connected via a bus or other means. Figure 14 Taking the example of a connection between China and Israel via a bus.

[0158] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the data sharing method in this embodiment of the invention. The processor 21 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the data sharing method described in the above embodiment.

[0159] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 22 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include memory remotely located relative to the processor 21, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] The one or more modules are stored in the memory 22. When executed by the one or more processors 21, they perform the data sharing method in any of the above method embodiments, for example, the method steps of the data sharing method described above.

[0161] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0162] Furthermore, the electronic device may also include components such as a Wi-Fi device 23, a display screen 24, a Bluetooth device 25, an audio circuit 26, a power supply system 27, a peripheral interface 28, a sensor module 29, and a data conversion module 30. These components can communicate via one or more communication buses or signal lines. Those skilled in the art will understand that... Figure 15 The hardware structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0163] The processor 21 is the control center of the electronic device, connecting various parts of the device through various interfaces and lines. It executes various functions and processes data by running or executing applications stored in the memory 22 and calling data and instructions stored in the memory 22. In some embodiments, the processor 21 may include one or more processing units; the processor 21 may also integrate an application processor 21 and a modem processor 21; wherein the application processor 21 mainly handles the operating system, user interface, and applications, and the modem processor 21 mainly handles wireless communication. It is understood that the modem processor 21 may also not be integrated into the processor 21.

[0164] In other embodiments of the present invention, the processor 21 may further include an Artificial Intelligence (AI) chip. The learning and processing capabilities of an AI chip include image understanding, natural language understanding, and speech recognition. AI chips can enable electronic devices to have better performance, longer battery life, and better security and privacy. For example, if an electronic device processes data through the cloud, the data needs to be uploaded for processing before the results are returned, which is inefficient under current technology. If the electronic device has strong AI learning capabilities locally, then the electronic device does not need to upload data to the cloud and can process it directly locally, thus improving processing efficiency while enhancing data security and privacy.

[0165] The memory 22 is used to store application programs and data. The processor 21 executes various functions and data processing of the electronic device by running the application programs and data stored in the memory 22. The memory 22 mainly includes a program storage area and a data storage area. The program storage area can store the operating system and application programs required for at least one function (such as sound playback function, image playback function, etc.); the data storage area can store data created when using the electronic device (such as audio data, video data, etc.). In addition, the memory 22 may include high-speed random access memory 22, and may also include non-volatile memory 22, such as disk storage devices 22, flash memory devices, or other non-volatile solid-state memory 22. The memory 22 can store various operating systems, such as the operating systems developed by Apple Inc., the operating systems developed by Microsoft Inc., etc.

[0166] Display screen 24 is used to display images, videos, etc. Display screen 24 may be a touchscreen. In some embodiments, the electronic device may include one or N displays screens 24, where N is a positive integer greater than 1. Processor 21 may include one or more graphics processing units (GPUs), which execute program instructions to generate or modify display information. The electronic device implements display functionality through the GPU and display screen 24. The GPU is used to perform mathematical and geometric calculations and for graphics rendering.

[0167] Wi-Fi device 23 provides network access to electronic devices that conforms to Wi-Fi related standard protocols. Electronic devices can access a Wi-Fi access point via Wi-Fi device 23, enabling them to browse web pages and access streaming media, providing users with wireless broadband internet access. Electronic devices can also establish Wi-Fi connections with terminal devices connected to the Wi-Fi access point via Wi-Fi device 23 and the Wi-Fi access point for mutual data transmission. In some other embodiments, Wi-Fi device 23 can also function as a Wi-Fi wireless access point, providing Wi-Fi network access to other electronic devices. Example, Wi-Fi device 23 includes at least one wireless network card.

[0168] Bluetooth device 25 is used to enable data exchange between electronic devices and other short-range electronic devices (such as terminals, smartwatches, etc.). In this embodiment of the invention, Bluetooth device 25 may be an integrated circuit or a Bluetooth chip, etc.

[0169] The audio circuit 26, speaker, and microphone provide an audio interface between the user and the electronic device. The audio circuit 26 converts the received audio data into electrical signals and transmits them to the speaker, where the speaker outputs them as sound signals. On the other hand, the microphone converts the collected sound signals into electrical signals, which are then received by the audio circuit 26, converted into audio data, and sent to the terminal via the Internet, Wi-Fi network, or Bluetooth, or output to the memory 22 for further processing.

[0170] The power system 27 is used to charge the various components of the electronic device. The power system 27 may include a battery and a power management module. The battery can be logically connected to the processor 21 through a power management chip, thereby enabling the power system 27 to manage functions such as charging, discharging, and power consumption.

[0171] Peripheral interface 28 provides various interfaces for external input / output devices (external display, external storage 22, user identification module card, etc.). For example, it connects to external storage 22, such as a microSD card, via the external storage 22 interface to expand the storage capacity of the electronic device. Peripheral interface 28 can be used to couple the aforementioned external input / output peripheral devices to processor 21 and memory 22.

[0172] Sensor module 29 may include at least one sensor, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor. The ambient light sensor can adjust the brightness of the display screen 24 according to the ambient light level. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity, which can be used for applications that identify the posture of electronic devices (such as landscape / portrait switching, related games, magnetometer posture calibration, etc.). Of course, depending on actual needs, sensor module 29 may also include any other feasible sensors.

[0173] The data conversion module 30 may include a digital-to-analog converter (DAC) and an analog-to-digital converter (ADC). The functions of the DAC and ADC are explained in the aforementioned technical terminology and will not be repeated here.

[0174] This invention also provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to perform the method steps of the data sharing method described above.

[0175] This invention also provides a computer program product, including a computing program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the data sharing method in any of the above method embodiments, for example, to perform the method steps of the data sharing method described above.

[0176] Based on the aforementioned electronic devices, embodiments of the present invention also provide a data sharing system, such as... Figure 16 As shown, the system includes a transmitting end and an electronic device. The transmitting end is used to connect to a terminal device (i.e., a personal terminal), and after acquiring the first streaming media data from the terminal device, sends the first streaming media data to the electronic device. The electronic device is used to establish a remote conference with a remote device through a conferencing application and execute the data sharing method described in the first embodiment. In practical application scenarios, the transmitting end can be a screen sharing device.

[0177] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above. For the sake of brevity, they are not provided in detail; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data sharing method applied to electronic devices, characterized in that, The method comprises: establishing a remote conference with at least one remote device through a conference application; receiving a first operation of a user; in response to the first operation, sharing a local picture to the remote device through the conference application; receiving first stream media data from a sending end; decoding the first stream media data to obtain second stream media data in a preset format; wherein the preset format is supported by a core layer of a native system of the electronic device; transmitting the second stream media data to a virtual driver program arranged in the core layer of the native system; the virtual driver program is identified as a virtual audio and video node by the conference application; after directly reading the second stream media data of the virtual audio and video node through the conference application, displaying the second stream media data and sharing the second stream media data to the remote device.

2. The data sharing method of claim 1, wherein, The method further comprises: detecting access of an audio and video device; when detecting the access of the audio and video device, shielding audio and video data sent by the audio and video device to the conference application until detecting that the electronic device disconnects the connection with the sending end or the conference application stops sharing to the remote device.

3. The data sharing method of claim 1, wherein, The method further comprises: receiving a second operation of a user; according to the second operation, superimposing annotation handwriting on a display window of the second stream media data; sharing the annotation handwriting to the remote device.

4. The data sharing method of claim 1, wherein, The sending end is a screen transmission device, and the first stream media data is desktop data sent by a personal terminal connected with the screen transmission device to the screen transmission device; or, the sending end is a screen transmission application installed on the personal terminal, and the first stream media data is desktop data of the personal terminal captured by the screen transmission application.

5. The data sharing method of claim 1, wherein, The transmitting the second stream media data to the virtual driver program arranged in the core layer of the native system specifically comprises: sending the second stream media data to a HAL layer through a preset interface, and then sending the second stream media data to the virtual driver program of the core layer by the HAL layer; wherein the virtual driver program comprises a virtual camera driver program and a virtual microphone driver program, and the virtual audio and video node comprises a virtual camera node and a virtual microphone node.

6. The data sharing method of claim 5, wherein, The method further comprises: monitoring whether there is access of a virtual camera driver program through a hot plug thread; the hot plug thread is used to monitor whether the virtual camera driver program is accessed; when monitoring that the virtual camera driver program is accessed through the hot plug thread, loading a corresponding virtual camera node to a camera hardware abstraction layer, so as to be identified by the conference application to obtain the virtual camera node.

7. The data sharing method of claim 6, wherein, The method further comprises: monitoring whether the accessed virtual camera driver program is closed through the hot plug thread; when monitoring that the virtual camera driver program is closed through the hot plug thread, removing the virtual camera node loaded to the camera hardware abstraction layer.

8. The data sharing method according to any one of claims 1-7, wherein, The format of the first stream media data is one or more of rtsp, rtmp, rtp or srtp; the preset format is one or more of YUV format, RGB format, JPEG format or PCM format.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data sharing method of any one of claims 1-8.

10. A non-transitory computer storage medium, comprising, The computer storage medium stores computer executable instructions executed by one or more processors to complete the data sharing method of any one of claims 1-8.

11. A data sharing system, characterized by, comprising a sending end and an electronic device; the sending end is configured to connect a terminal device, acquire first streaming media data of the terminal device, and send the first streaming media data to the electronic device; the electronic device is configured to establish a remote conference with a remote device through a conference application, and perform the data sharing method of any one of claims 1-8.

Images