Hardware device calling method and apparatus

Abstract

The present application provides a hardware device calling method and apparatus, for use in improving the compatibility, expansibility, and flexibility of a first terminal device calling hardware devices of a second terminal device. The method comprises: a first terminal device acquiring hardware device information of a second terminal device, the hardware device information comprising device identifier information of the second terminal device, type identifier information of at least one hardware device of the second terminal device, and hardware device capability information of the at least one hardware device; on the basis of the hardware device information and by means of a virtual bus driver, creating a virtual device driver for each hardware device among the at least one hardware device; and when detecting a message request for a first application program to call a first hardware device among the at least one hardware device by means of a first virtual device driver, receiving a data stream from the second terminal device by means of a data channel, the first virtual device driver being created by the virtual bus driver on the basis of hardware device information of the first hardware device.

Description

Hardware device calling method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 28, 2023, with application number 202310782520.1 and application name “Hardware Device Calling Method and Apparatus”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a method and apparatus for calling a hardware device. Background Art

[0003] Currently, for terminal devices that support interconnection services, such as mobile phones, tablets, laptops, etc., users can interconnect the first terminal device and the second terminal device through the interconnection service, thereby enabling the use of the hardware devices of the second terminal device, such as cameras, microphones, speakers, etc., through applications on the first terminal device.

[0004] For example, if the first terminal device is a laptop, the second terminal device is a mobile phone, the hardware device is a camera, and the application is a conferencing application, when a user uses a laptop for video conferencing and needs to perform an experimental demonstration, it is difficult to capture the entire experimental demonstration screen through the laptop's camera. The laptop can call the mobile phone's camera function to obtain video data captured by the mobile phone's camera, thereby improving the effect of the video demonstration. Specifically, the laptop includes a pre-developed virtual device driver, which uses the virtual device driver to call the mobile phone's camera to shoot and obtain the captured video data.

[0005] However, such hardware device calling methods have poor compatibility, scalability and flexibility.

[0006] Summary of the Invention

[0007] The present application provides a hardware device calling method and apparatus, which can improve the compatibility, scalability and flexibility of a hardware device called by a first terminal device on a second terminal device.

[0008] In a first aspect, a hardware device calling method is provided, which is applied to a system including a first terminal device and a second terminal device, wherein a user logs in to the same account on the first terminal device and the second terminal device, and the wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are both in the on state, the method comprising: when the second terminal device is detected, the first terminal device obtains the hardware device information of the second terminal device, the hardware device information including the device identification information of the second terminal device, the type identification information of at least one hardware device of the second terminal device, and the hardware device capability information of the at least one hardware device; based on the hardware device information, the first terminal device creates a virtual device driver for each hardware device in the at least one hardware device through a virtual bus driver, and the virtual bus driver is used to manage the virtual device driver; when it is detected that the first application calls a message request of the first hardware device in the at least one hardware device through the first virtual device driver, the first terminal device receives a data stream from the second terminal device through a data channel, and the first virtual device driver is created by the virtual bus driver based on the hardware device information of the first hardware device.

[0009] The hardware device calling method of the present application is that when a first terminal device detects a second terminal device, the first terminal device can obtain the hardware device information of the hardware device in the second terminal device; based on the hardware device information of each hardware device, the first terminal device can create a virtual device driver corresponding to each hardware device through a virtual bus driver; then, when the first terminal device needs to call the hardware device of the second terminal device, it can obtain the data stream from the second terminal device through the virtual device driver corresponding to the hardware device. It can be seen from this that the first terminal device can dynamically create virtual device drivers with different attributes based on the type and number of hardware devices of the second terminal device through the virtual bus driver. In this way, when the first terminal device calls the hardware devices other than the camera in the second terminal device, there is no need to re-develop the virtual device drivers for the hardware devices other than the camera, which improves the scalability of the hardware device calling method; and the virtual bus driver can create a virtual device driver corresponding to any USB hardware device, so that the first terminal device can call a variety of USB hardware devices, which improves the compatibility of the hardware device calling method; in addition, the first terminal device can create a corresponding number of virtual device drivers based on the number of hardware devices of the second terminal device, and when the first terminal device no longer needs to call the hardware device of the second terminal device, the first terminal device can also delete the virtual device driver through the virtual bus driver, and can also delete the virtual bus driver at the same time, which improves the flexibility of the hardware device calling method.

[0010] It should be understood that an account can refer to a personally registered account provided by different terminal device brands to ensure the user's device security. After the first and second terminal devices are logged into the same account, data sharing between the devices can be achieved. If the first and second terminal devices are logged into the same account and both the WiFi and Bluetooth switches of the first and second terminal devices are turned on, the first terminal device can detect the second terminal device.

[0011] In certain implementations of the first aspect, the first terminal device includes an application layer and a kernel layer; the application layer includes the first application, a first device virtualization module and a first communication service manager, the first device virtualization module is used to control the creation and deletion of virtual device drivers and process requests from applications, and the first communication service manager is used to perform data transmission between devices; the kernel layer includes the virtual bus driver, the virtual device driver and the driver development framework.

[0012] It should be understood that the number of virtual device drivers may be 0 or more. In the case where there are more than one virtual device drivers, the properties of any two of the multiple virtual device drivers may be different.

[0013] In certain implementations of the first aspect, the first terminal device obtains the hardware device information of the second terminal device, including: the first device virtualization module obtains the hardware device information; based on the hardware device information, the first terminal device creates a virtual device driver for each hardware device of the at least one hardware device through the virtual bus driver, including: the first device virtualization module creates a virtual device driver corresponding to the type identification information, device identification information and attributes indicated by the hardware device capability information of each hardware device through the virtual bus driver.

[0014] It should be understood that the first device virtualization module can be used to implement data transmission between the first terminal device and the second terminal device, and can control the virtual bus driver to create a virtual device driver. The first terminal device can obtain hardware device information from the second terminal device through the first virtualization module; and can control the virtual bus driver to create or delete virtual device drivers through the first virtualization module.

[0015] In certain implementations of the first aspect, the creation of a virtual device driver corresponding to the attributes indicated by the type identification information, device identification information, and hardware device capability information of each hardware device includes: the first device virtualization module converting the type identification information and device identification information of each hardware device into a hardware device type descriptor in a universal serial bus (USB) protocol, and converting the hardware device capability information of each hardware device into a hardware device configuration descriptor in the USB protocol; the first device virtualization module transmitting the hardware device type descriptor and the hardware device configuration descriptor to the virtual bus driver; and the virtual bus driver creating a virtual device driver corresponding to the hardware device type descriptor and the hardware device configuration descriptor.

[0016] It should be understood that each available hardware device in the second terminal device can be identified as a USB device by the first terminal device, and the first device virtualization module needs to create a corresponding virtual device driver for each hardware device in the second terminal device. The hardware device type descriptor can be understood as a device descriptor, and the hardware device configuration descriptor can be understood as a configuration descriptor. In addition to the hardware device type descriptor and the hardware device configuration descriptor, the first device virtualization module can also configure other types of descriptors for each hardware device, such as string descriptors. The virtual bus driver can create a virtual device driver corresponding to the descriptor of each hardware device.

[0017] In certain implementations of the first aspect, the second terminal device includes a second communication service manager, and the first communication service manager and the second communication service manager are used to implement data transmission between the first terminal device and the second terminal device; the first device virtualization module obtains the hardware device information, including: the first device virtualization module establishes a control channel; the first communication service manager sends a first request to the second communication service manager through the control channel, and the first request is used to request the hardware device information; the first communication service manager receives the hardware device information from the second communication service manager; the first communication service manager transmits the hardware device information to the first device virtualization module.

[0018] It should be understood that the second terminal device may be a mobile phone, tablet computer, or other terminal device. The second terminal device may be installed with the Android system, and the software structure of the second terminal device may include an application layer and an application framework layer. The application framework layer of the second terminal device may include a second communication service manager. The control channel may be a data transmission channel for transmitting control requests.

[0019] In certain implementations of the first aspect, after the first communication service manager transmits the hardware device information to the first device virtualization module, the method further includes: the first device virtualization module deleting the control channel.

[0020] It should be understood that when a control request needs to be transmitted between the first terminal device and the second terminal device, the first device virtualization module can create a control channel, for example, the first terminal device needs to obtain the hardware device information of the second terminal device through the device virtualization module; when the first terminal device does not need to send a control request to the second terminal device, the first device virtualization module can delete the control channel, for example, after the first terminal device has created a virtual device driver corresponding to each hardware device in the second terminal device, the first device virtualization module can determine that the first terminal device no longer needs to send a control request to the second terminal device within a certain period of time, and can delete the control channel.

[0021] In certain implementations of the first aspect, the first device virtualization module establishes a control channel, including: when the hardware device information cannot be read from the cloud server, the first device virtualization module establishes the control channel.

[0022] It should be understood that when the hardware device information of the second terminal device is stored in the cloud server, the first terminal device can obtain the hardware device information directly from the cloud server; when the cloud server does not store the hardware device information of the second terminal device, the first terminal device needs to obtain the hardware device information from the second terminal device. At this time, the first terminal device requires the second terminal device to send a control request for requesting hardware device information. Therefore, the first device virtualization module establishes a control channel.

[0023] In certain implementations of the first aspect, the method also includes: when a change is detected in the cache message queue corresponding to the first virtual device driver, the first device virtualization module reads a second request from the cache message queue, and the second request is used to request to obtain the data stream; the first terminal device receives the data stream from the second terminal device through a data channel, including: the first device virtualization module sends the data stream to the virtual bus driver based on the second request; the virtual bus driver matches and encapsulates the data stream and the second request in the cache message queue to obtain an encapsulated data packet, and feeds the encapsulated data packet back to the first application.

[0024] It should be understood that the first device virtualization module can monitor the cache message queue corresponding to each virtual device driver in real time. When there is a change in the cache message queue corresponding to the first virtual device driver, the first device virtualization module can read the request in the cache message queue corresponding to the first virtual device driver. The second request can be a data request for requesting the transmission of a data stream, such as the transmission of video data, audio data, etc. The cache message queue is a message queue used to cache message requests related to the first virtual device driver. Therefore, when the virtual bus driver receives the data stream from the first device virtualization module, it needs to match the second request corresponding to the data stream from the cache message queue and encapsulate the second request and the data stream.

[0025] In certain implementations of the first aspect, the method further includes: when it is detected that the second terminal device is offline, the first terminal device deletes the virtual device driver corresponding to the at least one hardware device through the virtual bus driver, and the second terminal device being offline indicates that the WiFi switch and / or Bluetooth switch of at least one device among the first terminal device and the second terminal device is in the off state, or, the first terminal device and the second terminal device are not logged in to the same account.

[0026] It should be understood that when the second terminal device goes offline, the first terminal device will no longer be able to detect the second terminal device. Therefore, the first terminal device can no longer call the hardware device of the second terminal device. The first terminal device can delete the virtual device driver corresponding to at least one hardware device through the virtual bus driver.

[0027] In certain implementations of the first aspect, the first terminal device deletes the virtual device driver corresponding to the at least one hardware device through the virtual bus driver, including: the first communication service manager detects that the second terminal device is offline, and notifies the first device virtualization module to delete the virtual device driver corresponding to the at least one hardware device; the first device virtualization module passes the deletion message of the virtual device driver corresponding to the at least one hardware device to the virtual bus driver; the virtual bus driver deletes the virtual device driver corresponding to the at least one hardware device based on the deletion message.

[0028] It should be understood that the first terminal device and the second terminal device are interconnected through the first communication service manager and the second communication service manager. If the first communication service manager cannot connect to the second communication service manager, the first communication service manager can determine that the second terminal device is offline. If the second terminal device is offline, the first communication service manager sends a message to the first device virtualization module, instructing the deletion of the virtual device driver corresponding to at least one hardware device. Based on the message, the first device virtualization module controls the virtual bus driver to delete the virtual device driver corresponding to the at least one hardware device.

[0029] In certain implementations of the first aspect, the first terminal device obtains hardware device information of the second terminal device, including: obtaining the hardware device information when a first switch button in a setting interface of the first terminal device and a second switch button in a setting interface of the second terminal device are in an on state, the first switch button is used for the user to select whether to turn on or off a function of calling the hardware device, and the second switch button is used for the user to select whether to turn on or off a function of calling the hardware device.

[0030] It should be understood that the setting interface can be a system setting interface of the terminal device or a setting interface of an application of the terminal device. When the first switch button and the second switch button are both in the on state, the first terminal device can call the hardware device of the second terminal device.

[0031] In certain implementations of the first aspect, the method further includes: when the first switch button and / or the second switch button is switched from an on state to an off state, the first terminal device deletes the virtual device driver corresponding to the at least one hardware device through the virtual bus driver.

[0032] It should be understood that when the first switch button and / or the second switch button is switched from the on state to the off state, the first terminal device will no longer be able to continue to call the hardware device of the second terminal device. Therefore, the first terminal device can delete the virtual device driver corresponding to at least one hardware device through the virtual bus driver.

[0033] In certain implementations of the first aspect, the method further includes: the first terminal device displays a settings interface, the settings interface includes the first switch button; based on the user's selection operation of the first switch button, the first switch button is adjusted to an on state or an off state.

[0034] It should be understood that the selection operation may be a click operation, a slide operation, etc., and the user's selection operation can switch the state of the first switch button.

[0035] In certain implementations of the first aspect, the first terminal device includes a second application; and the first terminal device installing a virtual bus driver includes: the first terminal device installing the virtual bus driver through the second application.

[0036] It should be understood that the second application can call the installation script and installation package of the virtual bus driver to install the virtual bus driver. For example, the second application can be PC Manager.

[0037] In certain implementations of the first aspect, before the first terminal device obtains the hardware device information of the second terminal device, the method also includes: the first terminal device detects whether a calling condition of the hardware device is met, and the calling condition includes at least one of the following: the device type of the second terminal device is a preset type; the first terminal device supports the ability to call the hardware device; or, the function of the first terminal device to call the hardware device is in the turned-on state; obtaining the hardware device information of the second terminal device includes: obtaining the hardware device information when the calling condition is met.

[0038] It should be understood that when a first terminal device discovers a second terminal device, the first terminal device can obtain the device type of the second terminal device. Preset types may include mobile phones, tablet computers, and the like. The first terminal device can call the hardware device of the preset type of the second terminal device. The first terminal device's ability to call hardware devices can be understood as the first terminal device having installed a virtual bus driver. The first terminal device's ability to call hardware devices being enabled can be understood as the first switch button being on.

[0039] In certain implementations of the first aspect, the first terminal device obtains the hardware device information of the second terminal device, including: the first terminal device reads the hardware device information from a cloud server.

[0040] It should be understood that when the cloud server stores the hardware device information of the second terminal device, the first terminal device can directly obtain the hardware device information from the cloud server, which can improve the efficiency of the first terminal device in obtaining the hardware device information of the second terminal device.

[0041] In some implementations of the first aspect, the at least one hardware device includes a camera and / or a microphone.

[0042] In a second aspect, the present application provides a hardware device invoking apparatus, comprising a processor coupled to a memory and configured to execute instructions in the memory to implement the method of any possible implementation of the first aspect described above. Optionally, the apparatus further comprises a memory. Optionally, the apparatus further comprises a communication interface, the processor coupled to the communication interface.

[0043] In one implementation, the apparatus is a first terminal device. When the apparatus is a first terminal device, the communication interface may be a transceiver, or an input / output interface.

[0044] In another implementation, the device is a chip configured in a terminal device. When the device is a chip configured in a terminal device, the communication interface may be an input / output interface.

[0045] In a third aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method of any possible implementation of the first aspect.

[0046] In a specific implementation process, the processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, or various logic circuits. The input signal received by the input circuit may be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit may be, for example, but not limited to, output to and transmitted by a transmitter. The input circuit and the output circuit may be the same circuit, which functions as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation of the processor and various circuits.

[0047] In a fourth aspect, a processing device is provided, comprising a processor and a memory. The processor is configured to read instructions stored in the memory and receive signals via a receiver and transmit signals via a transmitter to execute the method of any possible implementation of the first aspect.

[0048] Optionally, there are one or more processors and one or more memories.

[0049] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0050] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory and the setting method of the memory and the processor.

[0051] It should be understood that related data interaction processes, such as sending indication information, can be processes for outputting indication information from a processor, and receiving capability information can be processes for receiving input capability information from a processor. Specifically, the output data of the processor can be output to a transmitter, and the input data received by the processor can come from a receiver. The transmitter and receiver can be collectively referred to as a transceiver.

[0052] The processing device in the fourth aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0053] In a fifth aspect, a computer program product is provided, comprising: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute the method in any possible implementation of the first aspect.

[0054] In a sixth aspect, a computer-readable storage medium is provided, which stores a computer program (also referred to as code, or instructions) which, when run on a computer, enables the computer to execute the method in any possible implementation of the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0056] FIG2 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0057] FIG3 is a software structure block diagram of a Windows system of a terminal device according to an embodiment of the present application;

[0058] FIG4 is a flow chart of a hardware device calling method provided in an embodiment of the present application;

[0059] FIG5 is a schematic diagram of a process of a first application program acquiring data stream or hardware device information according to an embodiment of the present application;

[0060] FIG6 is a flow chart of another hardware device calling method provided in an embodiment of the present application;

[0061] FIG7 is a flow chart of another hardware device calling method provided in an embodiment of the present application;

[0062] FIG8 is a schematic diagram of an interface of a notebook computer display device manager provided in an embodiment of the present application;

[0063] FIG9 is a schematic diagram of an interface of a laptop computer displaying a second interface provided by an embodiment of the present application;

[0064] FIG10 is a schematic diagram of an interface of another notebook computer display device manager provided in an embodiment of the present application;

[0065] FIG11 is a schematic diagram of an interface of another laptop computer displaying a virtual device according to an embodiment of the present application;

[0066] FIG12 is a schematic block diagram of a hardware device calling apparatus provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solution in this application will be described below with reference to the accompanying drawings.

[0068] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first and second numerical values ​​are merely used to distinguish different numerical values ​​and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean different.

[0069] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0070] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0071] In order to better understand the terminal device in the embodiment of the present application, the hardware structure of the terminal device in the embodiment of the present application is described in detail below with reference to FIG1 .

[0072] Figure 1 is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of the present application. As shown in Figure 1, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0073] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0075] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0076] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0077] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0078] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the terminal device 100.

[0079] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0080] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0081] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0082] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0083] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0084] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal device 100 and to transfer data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect to other terminal devices, such as AR devices.

[0085] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.

[0086] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal device via the power management module 141.

[0087] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0088] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0089] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0090] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0091] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0092] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0093] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0094] The terminal device 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0095] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0096] The terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0097] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0098] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0099] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0100] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows terminal device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0101] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in the terminal device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0102] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0103] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0104] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0105] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0106] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0107] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0108] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and can also identify the source of sound, realize directional recording function, etc.

[0109] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0110] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, terminal device 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 100 can also calculate the location of the touch based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0111] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for shooting anti-shake. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0112] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0113] The magnetic sensor 180D includes a Hall effect sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Furthermore, based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0114] Accelerometer 180E can detect the magnitude of acceleration of the terminal device 100 in all directions (generally three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the terminal device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0115] The distance sensor 180F is used to measure distance. The terminal device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0116] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect when the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0117] Ambient light sensor 180L is used to sense ambient light brightness. Terminal device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether terminal device 100 is in a pocket to prevent accidental touches.

[0118] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0119] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to prevent the terminal device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0120] The touch sensor 180K is also called a "touch control device." The touch sensor 180K can be provided on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be provided on the surface of the terminal device 100, at a location different from that of the display screen 194.

[0121] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0122] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0123] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0124] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0125] The SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from the terminal device 100 by inserting or removing it from the SIM card interface 195. The terminal device 100 may support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 may support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. These cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as call and data communications. In some embodiments, the terminal device 100 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100. The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the terminal device 100.

[0126] Currently, for terminal devices that support interconnection services, such as mobile phones, tablets, laptops, etc., users can interconnect the first terminal device and the second terminal device through the interconnection service, thereby enabling the use of the hardware devices of the second terminal device, such as cameras, microphones, speakers, etc., through applications on the first terminal device.

[0127] 2 , the application scenario of the embodiment of the present application is described in detail by taking the first terminal device as a laptop, the second terminal device as a mobile phone, and the hardware device as a camera as an example.

[0128] FIG2 is a schematic diagram of an application scenario 200 provided in an embodiment of the present application. As shown in FIG2 , when a user is conducting a video conference on a laptop computer 201 through a video conferencing application, if it is necessary to photograph experimental equipment, the user can interconnect the laptop computer 201 and the mobile phone 202 so that data can be transmitted between the laptop computer 201 and the mobile phone 202. In this way, the laptop computer 201 can call the camera of the mobile phone 202, and the video data stream of the experimental equipment captured by the camera of the mobile phone 202 can be transmitted to the laptop computer 201, thereby displaying the experimental equipment to other people participating in the video conference.

[0129] Specifically, the first terminal device calling the hardware device of the second terminal device can currently be achieved through the following process: the application in the first terminal device can send a request to the virtual camera driver preset in the first terminal device, and the request is used to call the camera in the second terminal device; the first terminal device then obtains the video data captured by the camera of the second terminal device based on the request.

[0130] However, the virtual camera driver in the first terminal device is pre-set. Therefore, the amount of camera data that the first terminal device can recognize is fixed. If the number of cameras included in the second terminal device is greater than the number of virtual camera drivers, the first terminal device cannot fully recognize all the cameras of the second terminal device. Through the virtual camera driver, the first terminal device cannot call hardware devices other than the camera in the second terminal device, such as microphones, speakers, etc.; in addition, if the first terminal device does not need to call the camera of the second terminal device, the virtual camera driver will not be deleted. Therefore, this hardware device calling method has poor compatibility, scalability and flexibility.

[0131] To solve the above technical problems, the present application provides a hardware device calling method, which is applied to a system including a first terminal device and a second terminal device, wherein a user has logged into the same account on the first terminal device and the second terminal device, and the wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are both turned on. A virtual bus driver is installed in the first terminal device. When the first terminal device detects the second terminal device, the first terminal device can obtain hardware device information of the hardware devices in the second terminal device; based on the hardware device information of each hardware device, the first terminal device can create a virtual device driver corresponding to each hardware device through the virtual bus driver; then, when the first terminal device needs to call the hardware device of the second terminal device, it can obtain the data stream from the second terminal device through the virtual device driver corresponding to the hardware device. It can be seen from this that the first terminal device can dynamically create a virtual device driver based on the type and number of hardware devices of the second terminal device through the virtual bus driver. When the first terminal device calls the hardware devices other than the camera in the second terminal device, there is no need to redevelop the virtual device drivers for the hardware devices other than the camera, which improves the scalability of the hardware device calling method; and the virtual bus driver can create a virtual device driver corresponding to any USB hardware device, so that the first terminal device can call a variety of USB hardware devices, which improves the compatibility of the hardware device calling method; in addition, when the first terminal device no longer needs to call the hardware device of the second terminal device, the first terminal device can delete the virtual device driver based on the virtual bus driver, and can also delete the virtual bus driver at the same time, which improves the flexibility of the hardware device calling method.

[0132] 3 , the software structure applicable to the hardware device calling method according to the embodiment of the present application is described in detail.

[0133] Taking the terminal device 100 as a personal computer (PC) as an example, FIG3 is a software structure block diagram of the Windows system of the terminal device 100 according to an embodiment of the present application.

[0134] As shown in Figure 3, the layered architecture of the Windows system divides the software into several layers, each with clear roles and divisions of labor. Layers communicate with each other through software interfaces. For example, the application layer and the system layer can interact through the application programming interface (API), and the system layer and the kernel layer can interact through standard I / O interfaces. In some embodiments, the Windows system is divided into three layers: the application layer, the system layer, and the kernel layer, from top to bottom. The specific functional modules included in each layer are explained below.

[0135] 1. Application layer

[0136] The application layer may include a series of applications, such as music, PC manager, Bluetooth, WLAN, games, videos, third-party applications, cameras, software development kits (SDKs), communication service managers, device virtualization modules, device managers, etc. For example, in the embodiment of the present application, in the application layer, the application can access the device virtualization SDK and call the device virtualization module of the system layer through the device virtualization SDK.

[0137] The PC manager may include installation packages for various applications, and users may use the PC manager to install various applications for the terminal device 100. The PC manager is also used to implement functions such as system optimization, smart audio-visual, smart interconnection between devices, multi-screen collaboration, and hotspot sharing. The PC manager may also have other names, which are not specifically limited in this application.

[0138] Optionally, the PC manager includes a communication service manager and a device virtualization module, and the PC manager implements services such as smart interconnection through the communication service manager.

[0139] The above-mentioned third-party applications may include social applications and office applications.

[0140] The communication service manager can realize data transmission between devices. For example, in the embodiment of the present application, the communication service manager can realize security authentication and account authentication of devices with the same account, self-discovery and self-organizing networking of devices, data transmission between devices, and access to communication services.

[0141] The communication service manager can also be called a smart interconnection module, magiclink, etc. This application does not make any specific restrictions on this.

[0142] The device virtualization module manages the lifecycle of virtual device drivers, controlling their creation and deletion (also known as insertion and removal) through the virtual bus driver. It also processes requests from applications to virtual device drivers and provides feedback to the applications. The device virtualization module can interact with other applications through the device virtualization SDK at the application layer.

[0143] The device virtualization module may also be referred to as a hardware service cross-device sharing module, a distributed mobile sensing development platform (DMSDP), a hardware virtualization module, etc., and this application does not specifically limit this. Correspondingly, the device virtualization SDK may also be referred to as a hardware service cross-device sharing SDK, a DMSDP SDK, a hardware virtualization SDK, etc., and this application does not limit this either.

[0144] Device Manager is used to view and change device properties, change device drivers, configure device settings, and uninstall devices.

[0145] 2. System layer

[0146] The system layer may include the Windows multimedia framework, etc., which can provide corresponding services for the execution of user application processes.

[0147] The Windows Multimedia Framework includes a variety of Windows functions that can be called by applications for developing applications. The Windows Multimedia Framework is also used to connect the application layer and the kernel layer, enabling interaction between the application layer and the kernel layer. For example, requests sent by applications to the kernel layer can first be transmitted to the Windows Multimedia Framework through the Windows Application Framework (FWK) API, and then transmitted to the kernel layer through the standard I / O interaction interface.

[0148] 3. Kernel layer

[0149] The kernel layer is the layer between hardware and software. It drives the hardware, enabling it to operate. This layer can include virtual bus drivers, virtual device drivers, Windows Driver Foundation (WDF) modules, and the operating system (OS) kernel.

[0150] The virtual device driver can be identified by the terminal device as a universal serial bus (USB) device and used by the terminal device. For example, a virtual camera device, a virtual microphone (mic) device. If the terminal device 100 identifies an available USB device (also called a hardware device), the terminal device 100 can create a corresponding virtual device driver for each USB device. The number of virtual device drivers included in the kernel layer of the terminal device 100 can be multiple; if the terminal device 100 does not identify an available USB device, the number of virtual device drivers included in the kernel layer of the terminal device 100 is 0. In the case where the terminal device 100 creates at least one virtual device driver, the at least one virtual device driver can be displayed in the device manager of the terminal device 100.

[0151] The virtual bus driver is used to create or delete virtual device drivers, that is, it is responsible for managing the insertion, removal and communication processes of virtual device drivers.

[0152] Optionally, PC Manager can call the installation script and installation package of the virtual bus driver to install the virtual bus driver.

[0153] The virtual bus driver may also be called a virtual bus controller, a virtual driver bus, etc., which is not specifically limited in this application.

[0154] The OS kernel is used to convert requests from the application layer into data processing instructions and send them to the virtual device driver through the driver development framework.

[0155] The driver development framework is used to implement sending data processing instructions from the OS kernel to the virtual device driver.

[0156] The hardware device calling method of the present application is described in detail below with reference to Figures 4 to 11. The embodiment shown in this application illustrates the hardware device calling method provided by this application from the perspective of device interaction. The specific form and quantity of each device shown therein are only examples and should not constitute any limitation on the implementation of the method provided by this application. Below, taking the first terminal device and the second terminal device as the execution subjects as an example, the hardware device calling method of the embodiment of this application is described in detail.

[0157] It should be understood that the terminal device can be the terminal device itself, or it can be a chip, chip system or processor that supports the terminal device to implement the hardware device calling method, or it can be a logic module or software that can implement all or part of the terminal device functions. This application does not make specific limitations on this.

[0158] Figure 4 is a flow chart illustrating a method 400 for invoking a hardware device according to an embodiment of the present application. Method 400 is applied to a system comprising a first terminal device and a second terminal device, wherein a user has logged into the same account on both the first and second terminal devices, and both the wireless fidelity (WiFi) and Bluetooth switches on the first and second terminal devices are turned on. The hardware structure of the first terminal device may be as shown in Figure 1 , and the software structure may be as shown in Figure 3 .

[0159] It should be understood that an account can refer to a personally registered account provided by different terminal device brands to ensure the user's device security. After the first and second terminal devices are logged into the same account, data sharing between the devices can be achieved. If the first and second terminal devices are logged into the same account and both the WiFi and Bluetooth switches of the first and second terminal devices are turned on, the first terminal device can detect the second terminal device.

[0160] The method 400 includes the following steps:

[0161] S401. When a second terminal device is detected, obtain hardware device information of the second terminal device, where the hardware device information includes device identification information of the second terminal device, type identification information of at least one hardware device of the second terminal device, and hardware device capability information of at least one hardware device.

[0162] Optionally, before S401 , the first terminal device may install a virtual bus driver, where the virtual bus driver is used to manage virtual device drivers.

[0163] Specifically, the first terminal device may send a request for requesting the hardware device information of the second terminal device to the second terminal device; and the second terminal device may send the hardware device information of the second terminal device to the first terminal device based on the request.

[0164] Device identification information can be any information used to identify the second terminal device, such as the device type, device model, or device ID. For example, the device identification information is the device model, such as H90, which indicates that the second terminal device is a mobile phone with the model H90. The device identification information can also be the device name of the second terminal device set by the user, such as Zhang San's mobile phone. The type identification information indicates the type of at least one hardware device included in the second terminal device. Hardware devices can include cameras, microphones, speakers, and other devices. Different types of identification information can correspond to different hardware devices. For example, type identification information can include 0 and 1, where 0 can indicate the camera of the second terminal device and 1 can indicate the microphone of the second terminal device. Different types of identification information can correspond to different hardware devices of the same type. For example, type identification information can include A and B, where A can indicate the front camera of the second terminal device and B can indicate the rear camera of the second terminal device. Hardware device capability information can include at least one type of information used to indicate the performance of the hardware device. For example, if the hardware device is a camera, the hardware device capability information of the camera can include information such as the resolution, magnification, or frame rate supported by the camera.

[0165] S402: Based on the hardware device information, create a virtual device driver for each hardware device in the at least one hardware device through a virtual bus driver, where the virtual bus driver is used to manage the virtual device drivers.

[0166] It should be understood that the first terminal device can determine the number, type, and hardware device capabilities of the hardware devices included in the second terminal device based on the hardware device information. In this way, the first terminal device can create a corresponding virtual device driver for each hardware device based on the hardware device information of each hardware device in the at least one hardware device. Each hardware device in the second terminal device corresponds to a virtual device driver.

[0167] S403. When it is detected that the first application program calls a message request of a first hardware device among at least one hardware device through a first virtual device driver, the first terminal device receives a data stream from the second terminal device through a data channel, wherein the first virtual device driver is created by a virtual bus driver based on the hardware device information of the first hardware device.

[0168] It should be understood that the first application can be a third-party application or a system application. The first hardware device can be any hardware device in the second terminal device that can be used by the first terminal device, for example, it can be the front camera of the second terminal device. The hardware device information of the first hardware device includes the device identification information of the second terminal device, the type identification information of the first hardware device, and the hardware device capability information of the first hardware device. Based on the hardware device information of the first hardware device, the first terminal device can create a first virtual device driver with the same properties as the first hardware device through a virtual bus driver. The data channel can be understood as a channel that can be used to realize the transmission of data streams between the first terminal device and the second terminal device. For example, a peer-to-peer (P2P) connection is established between the first terminal device and the second terminal device, and the data channel can be a P2P connection.

[0169] In a possible implementation, before S403, method 400 further includes: the first terminal device establishing a data channel, wherein establishing a data channel can be understood as establishing a P2P connection.

[0170] The hardware device calling method of the present application is that when a first terminal device detects a second terminal device, the first terminal device can obtain the hardware device information of the hardware device in the second terminal device; based on the hardware device information of each hardware device, the first terminal device can create a virtual device driver corresponding to each hardware device through a virtual bus driver; then, when the first terminal device needs to call the hardware device of the second terminal device, it can obtain the data stream from the second terminal device through the virtual device driver corresponding to the hardware device. It can be seen from this that the first terminal device can dynamically create virtual device drivers with different attributes based on the type and number of hardware devices of the second terminal device through the virtual bus driver. In this way, when the first terminal device calls the hardware devices other than the camera in the second terminal device, there is no need to re-develop the virtual device drivers for the hardware devices other than the camera, which improves the scalability of the hardware device calling method; and the virtual bus driver can create a virtual device driver corresponding to any USB hardware device, so that the first terminal device can call a variety of USB hardware devices, which improves the compatibility of the hardware device calling method; in addition, the first terminal device can create a corresponding number of virtual device drivers based on the number of hardware devices of the second terminal device, and when the first terminal device no longer needs to call the hardware device of the second terminal device, the first terminal device can also delete the virtual device driver through the virtual bus driver, and can also delete the virtual bus driver at the same time, which improves the flexibility of the hardware device calling method.

[0171] As an optional embodiment, the first terminal device includes an application layer and a kernel layer; the application layer includes a first application, a first device virtualization module and a first communication service manager, the first device virtualization module is used to control the creation and deletion of virtual device drivers and process requests from applications, and the first communication service manager is used to transmit data between devices; the kernel layer includes a virtual bus driver, a virtual device driver and a driver development framework.

[0172] It should be understood that the software structure of the first terminal device can be seen in Figure 3 . The first application can be a third-party application or a system application at the application layer. The first device virtualization module can refer to the device virtualization module in Figure 3 ; the first communication service manager can refer to the communication service manager in Figure 3 . The number of virtual device drivers can be zero or more. If there are multiple virtual device drivers, the attributes of any two of the multiple virtual device drivers can be different.

[0173] In a possible implementation, the first terminal device creates a data channel, including: a first device virtualization module creates the data channel.

[0174] As an optional embodiment, S401 can be specifically implemented in the following manner: the first device virtualization module obtains hardware device information; S402 can be specifically implemented in the following manner: the first device virtualization module creates a virtual device driver corresponding to the attributes indicated by the type identification information, device identification information and hardware device capability information of each hardware device through a virtual bus driver.

[0175] It should be understood that the first device virtualization module can be used to implement data transmission between the first terminal device and the second terminal device, and can control the virtual bus driver to create a virtual device driver. The first terminal device can obtain hardware device information from the second terminal device through the first virtualization module; and can control the virtual bus driver to create or delete virtual device drivers through the first virtualization module.

[0176] In one possible implementation, a virtual device driver corresponding to the attributes indicated by the type identification information, device identification information, and hardware device capability information of each hardware device is created, including: a first device virtualization module converts the type identification information and device identification information of each hardware device into a hardware device type descriptor in a universal serial bus (USB) protocol, and converts the hardware device capability information of each hardware device into a hardware device configuration descriptor in the USB protocol; the first device virtualization module transmits the hardware device type descriptor and the hardware device configuration descriptor to a virtual bus driver; and the virtual bus driver creates a virtual device driver corresponding to the hardware device type descriptor and the hardware device configuration descriptor.

[0177] It should be understood that each available hardware device in the second terminal device can be identified as a USB device by the first terminal device, and the first device virtualization module needs to create a corresponding virtual device driver for each hardware device in the second terminal device. The hardware device type descriptor can be understood as a device descriptor, and the hardware device configuration descriptor can be understood as a configuration descriptor. In addition to the hardware device type descriptor and the hardware device configuration descriptor, the first device virtualization module can also configure other types of descriptors for each hardware device, such as string descriptors. The virtual bus driver can create a virtual device driver corresponding to the descriptor of each hardware device.

[0178] It should be understood that the hardware device type descriptor may include information other than the descriptors corresponding to the type identification information and the device identification information; the hardware device configuration descriptor may include information other than the descriptors corresponding to the hardware device capability information, and this application does not make specific limitations on this.

[0179] In one possible embodiment, the second terminal device includes a second communication service manager, and the first communication service manager and the second communication service manager are used to implement data transmission between the first terminal device and the second terminal device; the first device virtualization module obtains hardware device information, including: the first device virtualization module establishes a control channel; the first communication service manager sends a first request to the second communication service manager through the control channel, and the first request is used to request hardware device information; the first communication service manager receives hardware device information from the second communication service manager; the first communication service manager transmits the hardware device information to the first device virtualization module.

[0180] It should be understood that the second terminal device may be a mobile phone, tablet computer, or other terminal device. The second terminal device may be installed with the Android system, and the software structure of the second terminal device may include an application layer and an application framework layer. The application framework layer of the second terminal device may include a second communication service manager. The control channel may be a data transmission channel for transmitting control requests.

[0181] In a possible implementation, the control channel is created by establishing a P2P connection or a Bluetooth connection between the first terminal device and the second terminal device.

[0182] In one possible implementation, the control channel is preferentially created by establishing a P2P connection between the first terminal device and the second terminal device. If a P2P connection cannot be established between the first terminal device and the second terminal device, the control channel is created by establishing a Bluetooth connection between the first terminal device and the second terminal device.

[0183] In a possible implementation, after the first communication service manager transmits the hardware device information to the first device virtualization module, the method 400 further includes: the first device virtualization module deleting the control channel.

[0184] It should be understood that when a control request needs to be transmitted between the first terminal device and the second terminal device, the first device virtualization module can create a control channel, for example, the first terminal device needs to obtain the hardware device information of the second terminal device through the device virtualization module; when the first terminal device does not need to send a control request to the second terminal device, the first device virtualization module can delete the control channel, for example, after the first terminal device has created a virtual device driver corresponding to each hardware device in the second terminal device, the first device virtualization module can determine that the first terminal device no longer needs to send a control request to the second terminal device within a certain period of time, and can delete the control channel.

[0185] In a possible implementation, after the virtual device driver is created, the first device virtualization module waits for a preset period of time before deleting the control channel.

[0186] It should be understood that the control channel is deleted with a delay. Deleting the control channel can also be understood as destroying the control channel.

[0187] In a possible implementation, the first device virtualization module establishes the control channel, including: when the hardware device information cannot be read from the cloud server, the first device virtualization module establishes the control channel.

[0188] It should be understood that when the hardware device information of the second terminal device is stored in the cloud server, the first terminal device can obtain the hardware device information directly from the cloud server; when the cloud server does not store the hardware device information of the second terminal device, the first terminal device needs to obtain the hardware device information from the second terminal device. At this time, the first terminal device requires the second terminal device to send a control request for requesting hardware device information. Therefore, the first device virtualization module establishes a control channel.

[0189] As an optional embodiment, S401 is specifically implemented in the following manner: reading hardware device information from a cloud server.

[0190] It should be understood that when the cloud server stores the hardware device information of the second terminal device, the first terminal device can directly obtain the hardware device information from the cloud server, which can improve the efficiency of the first terminal device in obtaining the hardware device information of the second terminal device.

[0191] In a possible implementation, when a terminal device obtains hardware device information from a second terminal device, the second terminal device transmits the hardware device information to a cloud server.

[0192] It should be understood that the terminal device may be the first terminal device or another terminal device. In the event that a terminal device obtains hardware device information from a second terminal device at a historical moment, the second terminal device may determine that a terminal device needs to obtain the hardware device information, and therefore, the second terminal device may upload the hardware device information to the cloud server.

[0193] As an optional embodiment, method 400 also includes: when a change is detected in the cache message queue corresponding to the first virtual device driver, the first device virtualization module reads a second request from the cache message queue, and the second request is used to request to obtain a data stream; the first terminal device receives the data stream from the second terminal device through the data channel, including: the first device virtualization module sends the data stream to the virtual bus driver based on the second request; the virtual bus driver matches and encapsulates the data stream and the second request in the cache message queue to obtain an encapsulated data packet, and feeds the encapsulated data packet back to the first application.

[0194] It should be understood that the first device virtualization module can monitor the cache message queue corresponding to each virtual device driver in real time. When there is a change in the cache message queue corresponding to the first virtual device driver, the first device virtualization module can read the request in the cache message queue corresponding to the first virtual device driver. The second request can be a data request for requesting the transmission of a data stream, such as the transmission of video data, audio data, etc. The cache message queue is a message queue for caching message requests related to the first virtual device driver. The cache message queue can include multiple requests. Therefore, when the virtual bus driver receives the data stream from the first device virtualization module, it needs to match the second request corresponding to the data stream from the cache message queue and encapsulate the second request and the data stream.

[0195] In one possible implementation, when the first application sends a control request, the kernel layer of the first terminal device creates a control endpoint. The control endpoint can cache a control message queue, which can include multiple control requests. When the first application sends a data request, the kernel layer of the first terminal device creates a data endpoint. The data endpoint can cache a data message queue, which can include multiple control requests. The control endpoint can cache received control requests in the cache message queue, and the data endpoint can cache received data requests in the cache message queue.

[0196] Figure 5 is a schematic diagram of the process by which a first application program obtains data stream or hardware device information, as provided in an embodiment of the present application. As shown in Figure 5 , when the driver development framework detects a control request from the first application program, the driver development framework creates a control endpoint; when the driver development framework detects a data request from the first application program, the driver development framework creates a data endpoint. The control endpoint includes a control message queue; the data endpoint includes a data message queue. When the request sent by the first application program is a control request, such as the first request, and the driver development framework detects a change in the control message queue, the virtual bus driver caches the first request in the cache message queue. When the request sent by the first application program is a data request, such as the second request, and the driver development framework detects a change in the data message queue, the virtual bus driver caches the second request in the cache message queue. When the first device virtualization module detects a change in the cache message queue, it reads the requests in the cache message queue, such as the first and second requests described above. The first device virtualization module processes the read requests and sends request replies to the virtual bus driver. When the request read by the first device virtualization module is the first request, the request reply may be hardware device information of the second terminal device; when the request read by the first device virtualization module is the second request, the request reply may be a data stream. The virtual bus driver matches the request reply with the requests in the cached message queue and encapsulates it to obtain an encapsulated data packet. For example, it matches and encapsulates the hardware device information with the first request and the data stream with the second request. The virtual bus driver then responds to the first application's request by feeding the encapsulated data packet back to the first application.

[0197] As an optional embodiment, method 400 also includes: when it is detected that the second terminal device is offline, the first terminal device deletes the virtual device driver corresponding to at least one hardware device through the virtual bus driver. The second terminal device being offline indicates that the WiFi switch and / or Bluetooth switch of at least one device in the first terminal device and the second terminal device is in the off state, or the first terminal device and the second terminal device are not logged in to the same account.

[0198] It should be understood that when the second terminal device goes offline, the first terminal device will no longer be able to detect the second terminal device. Therefore, the first terminal device can no longer call the hardware device of the second terminal device, and the first terminal device can delete the virtual device driver corresponding to at least one hardware device through the virtual bus driver. The WiFi switch and / or Bluetooth switch of at least one of the first terminal device and the second terminal device being in the off state can refer to the WiFi switch and / or Bluetooth switch of at least one of the first terminal device and the second terminal device being switched from the on state to the off state; the fact that the first terminal device and the second terminal device are not logged in to the same account can be understood as the account logged in by the first terminal device and / or the second terminal device being offline.

[0199] In a possible real-time method, the first terminal device deletes the virtual device driver corresponding to at least one hardware device through a virtual bus driver, including: the first communication service manager detects that the second terminal device is offline, and notifies the first device virtualization module to delete the virtual device driver corresponding to the at least one hardware device; the first device virtualization module passes the deletion message of the virtual device driver corresponding to the at least one hardware device to the virtual bus driver; the virtual bus driver deletes the virtual device driver corresponding to the at least one hardware device based on the deletion message.

[0200] It should be understood that the first terminal device and the second terminal device are interconnected through the first communication service manager and the second communication service manager. If the first communication service manager cannot connect to the second communication service manager, the first communication service manager can determine that the second terminal device is offline. If the second terminal device is offline, the first communication service manager sends a message to the first device virtualization module, instructing the deletion of the virtual device driver corresponding to at least one hardware device. Based on the message, the first device virtualization module controls the virtual bus driver to delete the virtual device driver corresponding to the at least one hardware device.

[0201] In a possible implementation, the virtual bus driver deletes the virtual device driver corresponding to at least one hardware device based on the deletion message, including: the virtual bus driver deletes the descriptor corresponding to at least one hardware device based on the deletion message, the descriptor including the hardware device type descriptor and the hardware device configuration descriptor.

[0202] As an optional embodiment, S401 can be implemented in the following manner: when the first switch button in the first terminal device setting interface and the second switch button in the second terminal device setting interface are in the on state, the hardware device information is obtained, the first switch button is used for the user to choose to turn on or off the function of calling the hardware device, and the second switch button is used for the user to choose to turn on or off the function of the hardware device being called.

[0203] It should be understood that the setting interface can be a system setting interface of the terminal device or a setting interface of an application of the terminal device. When the first switch button and the second switch button are both in the on state, the first terminal device can call the hardware device of the second terminal device.

[0204] As an optional embodiment, method 400 further includes: when the first switch button and / or the second switch button is switched from an on state to an off state, the first terminal device deletes a virtual device driver corresponding to at least one hardware device through a virtual bus driver.

[0205] It should be understood that when the first switch button and / or the second switch button is switched from the on state to the off state, the first terminal device will no longer be able to continue to call the hardware device of the second terminal device. Therefore, the first terminal device can delete the virtual device driver corresponding to at least one hardware device through the virtual bus driver.

[0206] As an optional embodiment, method 400 further includes: the first terminal device displays a setting interface, the setting interface includes a first switch button; based on the user's selection operation of the first switch button, the first switch button is adjusted to an on state or an off state.

[0207] It should be understood that the selection operation may be a click operation, a slide operation, etc., and the user's selection operation can switch the state of the first switch button.

[0208] As an optional embodiment, the first terminal device includes a second application program; and the first terminal device installs a virtual bus driver, comprising: the first terminal device installs the virtual bus driver through the second application program.

[0209] It should be understood that the second application can call the installation script and installation package of the virtual bus driver to install the virtual bus driver. For example, the second application can be the PC manager shown in FIG. 3 .

[0210] In a possible implementation, when the second application is installed on the first terminal device, the second application calls the installation script and installation package of the virtual bus driver to install the virtual bus driver.

[0211] In a possible implementation, the user installs the virtual bus driver through an installation package of the virtual bus driver.

[0212] As an optional embodiment, before S401, method 400 also includes: the first terminal device detects whether the calling conditions of the hardware device are met, and the calling conditions include at least one of the following: the device type of the second terminal device is a preset type; the first terminal device supports the ability to call the hardware device; or the function of the first terminal device to call the hardware device is turned on; obtaining the hardware device information of the second terminal device, including: obtaining the hardware device information when the calling conditions are met.

[0213] It should be understood that when a first terminal device discovers a second terminal device, the first terminal device can obtain the device type of the second terminal device. Preset types may include mobile phones, tablet computers, and the like. The first terminal device can call the hardware device of the preset type of the second terminal device. The first terminal device's ability to call hardware devices can be understood as the first terminal device having installed a virtual bus driver. The first terminal device's ability to call hardware devices being enabled can be understood as the first switch button being on.

[0214] As an optional embodiment, the at least one hardware device includes a camera and / or a microphone.

[0215] Figure 6 is a flowchart of a hardware device invocation method 600 provided in an embodiment of the present application. As shown in Figure 6, method 600 is applied to a system including a first terminal device and a second terminal device, wherein a user has logged into the same account on the first terminal device and the second terminal device, and both the WiFi and Bluetooth switches of the first and second terminal devices are turned on.

[0216] The hardware structure of the first terminal device may be as shown in Figure 1, and the software structure may be as shown in Figure 3. The hardware structure of the second terminal device may be as shown in Figure 1, and the software structure of the second terminal device includes a second communication service manager and a second device virtualization module. The second device virtualization module is used to interact with the camera API of the first terminal device, and can turn on or off the camera of the first terminal device, obtain video data captured by the camera, and process the video data, such as encoding and decoding images; it is also used for the management of the control channel and the data channel, and the data channel is used to transmit data streams; it is used to interact with the application of the second terminal device to trigger the pause, resume, and close of the virtual camera.

[0217] The method 600 includes the following steps:

[0218] S601. Use PC Manager to install the virtual bus driver.

[0219] S602: The first device virtualization module obtains hardware device information of the second terminal device from the second device virtualization module.

[0220] Specifically, the second device virtualization module sends the hardware device information to the second communication service manager; the second communication service manager sends the hardware device information to the first communication service manager; and the first communication service manager sends the hardware device information to the first device virtualization module.

[0221] S603: The first device virtualization module notifies the virtual bus driver to insert the virtual device. This can be understood as the first device virtualization module writing the descriptor corresponding to the hardware device into the virtual bus driver.

[0222] S604: The virtual bus driver creates a virtual device driver for each hardware device in the second terminal device.

[0223] S605: The first device virtualization module reads the message in the cache message queue corresponding to the virtual device driver through the virtual bus driver.

[0224] S606: The first device virtualization module writes data to the virtual bus driver.

[0225] It should be understood that data can be understood as hardware device information.

[0226] S607: When the second terminal device is offline or the first terminal device no longer needs to call the hardware device of the second terminal device, the first device virtualization module notifies the virtual bus driver to unplug the virtual device. This can be understood as the first device virtualization module sending a deletion message to the virtual bus driver.

[0227] S608: The virtual bus driver deletes the virtual device driver.

[0228] S609: Uninstall the virtual bus driver through PC Manager.

[0229] It should be understood that if the user needs to delete the virtual bus driver, the virtual bus driver can be deleted through PC Manager.

[0230] It should be understood that the specific implementation of S601 to S609 is similar to the steps in method 400, and reference can be made to the above, which will not be repeated here.

[0231] 7 , the hardware device calling method is described in detail by taking a camera as an example.

[0232] Figure 7 is a flowchart of a hardware device call method 700 provided in an embodiment of the present application. As shown in Figure 7, method 700 is applied to a system including a first terminal device and a second terminal device, wherein a user has logged into the same account on the first terminal device and the second terminal device, and both the WiFi and Bluetooth switches of the first and second terminal devices are turned on.

[0233] The hardware structure of the first terminal device can be shown in Figure 1, and the software structure can be shown in Figure 3. The virtual bus driver in the first terminal device can be installed using PC Manager. After the virtual bus driver is installed using PC Manager, the virtual bus driver can be displayed in the device manager of the first terminal device. For example, as shown in Figure 8, the first terminal device is a laptop computer, and the system devices in the device manager of the laptop computer include the virtual bus driver.

[0234] The hardware structure of the second terminal device can be as shown in Figure 1. The software structure of the second terminal device includes a second communication service manager and a second device virtualization module. The second device virtualization module is used to interact with the camera API of the first terminal device, and can turn on or off the camera of the first terminal device, obtain video data captured by the camera, and process the video data, such as encoding and decoding images; it is also used for the management of control channels and data channels, and the data channel is used to transmit data streams; it is used to interact with the application of the second terminal device to trigger the pause, reply and close of the virtual camera.

[0235] Method 700 includes the following steps:

[0236] S701: The second communication service manager sends a first message to the first communication service manager, where the first message is used to notify the first communication service manager that a second terminal device is online. Correspondingly, the first communication service manager receives the first message from the second communication service manager.

[0237] Optionally, when the second terminal device is close to the first terminal device so that the distance between the second terminal device and the first terminal device is less than a preset distance, the second communication service manager sends a first message to the first communication service manager.

[0238] S702: The first communication service manager discovers an online device based on the first message. The online device can be understood as a second terminal device.

[0239] S703: The first communication service manager sends a second message to the application, where the second message is used to notify the application that the second terminal device is online. Correspondingly, the application receives the second message from the first communication service manager.

[0240] It should be understood that the second message may include information such as the device type and device model of the second terminal device. The application may be the PC manager in FIG3 .

[0241] S704: PC Manager discovers the online device based on the second message.

[0242] Optionally, PC Manager can discover online devices based on Smart Connect. If PC Manager can discover the second terminal device based on Smart Connect, the first terminal device displays a second interface, which includes the device type and model of the second terminal device. For example, if the first terminal device is a laptop, the second terminal device's model is H90, and its device type is a mobile phone, the second interface may be as shown in Figure 9.

[0243] S705: PC Manager checks whether the calling conditions of the hardware device are met.

[0244] It should be understood that PC Manager can detect whether the first terminal device and / or the second terminal device meet the hardware device call conditions. For example, PC Manager detects whether the device type of the first terminal device is a preset type; detects whether the first terminal device and / or the second terminal device support the virtual camera capability; detects whether the first switch button and the second switch button are turned on, etc.

[0245] If the first terminal device and the second terminal device meet the calling conditions of the hardware device, S706 is executed.

[0246] S706: PC Manager launches the first device virtualization module.

[0247] S707: PC Manager performs a registration callback on the first device virtualization module.

[0248] S708 : The PC manager notifies the first device virtualization module to enable the camera.

[0249] S709. The first device virtualization module obtains hardware device information from the cloud server.

[0250] S710: The first device virtualization module pulls up the second device virtualization module and creates a control channel.

[0251] Optionally, when the hardware device information is not stored in the cloud server, the first device virtualization module sends a first request to the second device virtualization module through the control channel; the second device virtualization module sends the hardware device information to the first communication service manager through the second communication service manager based on the first request; the first communication service manager sends the hardware device information to the first device virtualization module.

[0252] S711: The first device virtualization module instructs the virtual bus driver to create a virtual device driver based on the hardware device information.

[0253] S712. Create a virtual device driver based on the virtual bus driver.

[0254] Optionally, after the virtual bus driver creates the virtual device driver, the virtual device driver can be displayed in the device manager of the first terminal device. For example, the available hardware devices of the second terminal device include a front camera and a rear camera. The device model of the second terminal device is H90. As shown in FIG10 , after the virtual bus driver creates the virtual device driver corresponding to the front camera and the virtual device driver corresponding to the rear camera, the device manager of the first terminal device displays the virtual devices corresponding to the two virtual device drivers: H90 rear camera 1001 and H90 front camera 1002.

[0255] S713: The first device virtualization module reads a third message from the virtual bus driver. The third message is used to notify the first device virtualization module that the virtual device driver creation is complete.

[0256] S714 : The first device virtualization module sends a fourth message to the PC manager, where the fourth message is used to notify the PC manager that the virtual device driver creation is complete.

[0257] It should be understood that after the virtual device driver is created, the virtual device corresponding to the virtual device driver can be displayed in the first application. For example, as shown in Figure 11, the first application is a conference application, the hardware devices of the second terminal device include a front camera and a rear camera, and the device model of the second terminal device is H90. After the virtual bus driver creates the virtual device drivers corresponding to the front camera and the rear camera, the camera selection interface of the first application includes the virtual devices corresponding to the two virtual device drivers: H90 rear camera 1101 and H90 front camera 1102.

[0258] Optionally, the third-party application of the first terminal device sends the second request to the cache message queue corresponding to the virtual device driver through the Windows multimedia framework and the driver development framework in sequence. After detecting changes in the cache message queue corresponding to the virtual device driver, the first device virtualization module reads the second request; based on the second request, the first device virtualization module obtains the data stream from the second device virtualization module through the first communication service manager and the second communication service manager in sequence; the first device virtualization module sends the data stream to the virtual bus driver, and the virtual bus driver matches and encapsulates the data stream and the second request, and feeds back the encapsulated data packet to the third-party application. Exemplarily, the third-party application can be a conference application; the virtual device driver can be the driver corresponding to the front camera of the second terminal device and the driver corresponding to the rear camera of the second terminal device. When the third-party application calls the front camera, it sends the second request to the cache message queue of the driver corresponding to the front camera; then the first device virtualization module transmits the video data stream obtained by the front camera to the third-party application.

[0259] Optionally, after reading the second request, the first device virtualization module can determine that the third-party application needs to call the first hardware device of the second terminal device; send an instruction to open the first hardware device to the second device virtualization module through the first communication service manager and the second communication service manager in sequence; the second device virtualization module obtains the data stream obtained by the operation of the first hardware device based on the instruction, and sends the data stream obtained by the operation of the first hardware device to the first device virtualization module through the second communication service manager and the first communication service manager in sequence; the first device virtualization module sends the data stream obtained by the operation of the first hardware device to the virtual bus driver, the virtual bus driver matches and encapsulates the data stream obtained by the operation of the first hardware device and the second request, and feeds back the encapsulated data packet to the third-party application.

[0260] S715. The first device virtualization module deletes the control channel.

[0261] It should be understood that after the second device virtualization module detects that all control channels have been deleted, it triggers unbinding, meaning that control requests are no longer transmitted. After the second device virtualization module detects that all services have been unbound, it triggers deregistration. Unbinding all services can be understood as the first terminal device no longer calling the hardware devices of the second terminal device.

[0262] It should be understood that the present application does not limit the order of S715 and the process of the third-party application obtaining the data stream. S715 can be executed before, after or during the process of the third-party application obtaining the data stream.

[0263] S716: After the second communication service manager detects that the second terminal device is away from the first terminal device, it determines that the second terminal device is offline.

[0264] It should be understood that the second terminal device is far away from the first terminal device, which can be understood as the distance between the second terminal device and the first terminal device is too far, so that the second terminal device and the first terminal device are not in the same local area network, or the Bluetooth connection between the first terminal device and the second terminal device is disconnected.

[0265] S717: The first communication service manager sends a fifth message to the application, where the fifth message is used to notify the application that the second terminal device is offline. Correspondingly, the application receives the fifth message from the first communication service manager.

[0266] S718: The application program notifies the first device virtualization module to disable the camera.

[0267] S719: The first device virtualization module sends a deletion message to the virtual bus driver, where the deletion message is used to instruct the virtual bus driver to delete the virtual device driver. Correspondingly, the virtual bus driver receives the deletion message from the first device virtualization module.

[0268] S720: The virtual bus driver deletes the virtual device driver.

[0269] It should be understood that the specific implementation of S701 to S720 is similar to the steps in method 400, and reference can be made to the above, which will not be repeated here.

[0270] It should be understood that the order of execution of the above methods does not necessarily mean the order in which they are executed. The order in which the methods are executed should be determined by their functions and internal logic.

[0271] The above describes in detail the hardware device calling method of the embodiment of the present application in combination with Figures 2 to 11. The following describes in detail the hardware device calling apparatus of the embodiment of the present application in combination with Figure 12.

[0272] Figure 12 is a schematic block diagram of a hardware device invocation apparatus 1200 provided in an embodiment of the present application. Apparatus 1200 includes a processor 1201, a transceiver 1202, and a memory 1203. Processor 1201, transceiver 1202, and memory 1203 communicate with each other via an internal connection path. Memory 1203 is used to store instructions, and processor 1201 is used to execute the instructions stored in memory 1203 to control transceiver 1202 to send and / or receive signals.

[0273] It should be understood that apparatus 1200 can be specifically the first terminal device in the above-described embodiment, and can be used to execute the various steps and / or processes corresponding to the first terminal device in the above-described method embodiment. Optionally, the memory 1203 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 1201 can be used to execute instructions stored in the memory, and when the processor 1201 executes the instructions stored in the memory, the processor 1201 is used to execute the various steps and / or processes of the above-described method embodiment. The transceiver 1202 may include a transmitter and a receiver. The transmitter can be used to implement the various steps and / or processes corresponding to the above-described transceiver for performing a sending action, and the receiver can be used to implement the various steps and / or processes corresponding to the above-described transceiver for performing a receiving action.

[0274] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0275] During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0276] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method shown in the above method embodiment.

[0277] The present application also provides a computer program product, which includes a computer program (also referred to as code or instructions). When the computer program runs on a computer, the computer can execute the method shown in the above method embodiment.

[0278] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0279] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0280] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0281] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0282] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0283] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0284] The above description is merely a specific embodiment of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any person skilled in the art can easily conceive of changes or substitutions within the technical scope disclosed in the embodiments of the present application, and such changes or substitutions should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A method for calling a hardware device, characterized in that: Applied to a system including a first terminal device and a second terminal device, a user logs in to the same account on the first terminal device and the second terminal device, and both the wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are turned on, the method comprising: When the second terminal device is detected, the first terminal device acquires hardware device information of the second terminal device, where the hardware device information includes device identification information of the second terminal device, type identification information of at least one hardware device of the second terminal device, and hardware device capability information of the at least one hardware device; Based on the hardware device information, the first terminal device creates a virtual device driver for each hardware device in the at least one hardware device through a virtual bus driver, wherein the virtual bus driver is used to manage the virtual device driver; When it is detected that the first application calls a message request of the first hardware device among the at least one hardware device through the first virtual device driver, the first terminal device receives the data stream from the second terminal device through the data channel, and the first virtual device driver is created by the virtual bus driver based on the hardware device information of the first hardware device.

2. The method according to claim 1, characterized in that: The first terminal device comprises an application layer and a kernel layer; The application layer includes the first application, a first device virtualization module and a first communication service manager, the first device virtualization module is used to control the creation and deletion of virtual device drivers and process requests from the application, and the first communication service manager is used to perform data transmission between devices; The kernel layer includes the virtual bus driver, the virtual device driver and the driver development framework.

3. The method according to claim 2, characterized in that The first terminal device acquires the hardware device information of the second terminal device, including: The first device virtualization module obtains the hardware device information; The first terminal device creates a virtual device driver for each hardware device in the at least one hardware device through the virtual bus driver based on the hardware device information, including: The first device virtualization module creates a virtual device driver corresponding to the type identification information, the device identification information, and the attributes indicated by the hardware device capability information of each hardware device through the virtual bus driver.

4. The method according to claim 3, characterized in that The creating of a virtual device driver corresponding to the type identification information, the device identification information, and the attributes indicated by the hardware device capability information of each hardware device includes: The first device virtualization module converts the type identification information and the device identification information of each hardware device into a hardware device type descriptor in a universal serial bus (USB) protocol, and converts the hardware device capability information of each hardware device into a hardware device configuration descriptor in the USB protocol; The first device virtualization module transmits the hardware device type descriptor and the hardware device configuration descriptor to the virtual bus driver; The virtual bus driver creates a virtual device driver corresponding to the hardware device type descriptor and the hardware device configuration descriptor.

5. The method according to claim 3 or 4, characterized in that: The second terminal device comprises a second communication service manager, and the first communication service manager and the second communication service manager are used to implement data transmission between the first terminal device and the second terminal device; The first device virtualization module obtains the hardware device information, including: The first device virtualization module establishes a control channel; The first communication service manager sends a first request to the second communication service manager through the control channel, where the first request is used to request the hardware device information; The first communication service manager receives the hardware device information from the second communication service manager; The first communication service manager transmits the hardware device information to the first device virtualization module.

6. The method according to claim 5, characterized in that After the first communication service manager transmits the hardware device information to the first device virtualization module, the method further includes: The first device virtualization module deletes the control channel.

7. The method according to claim 5 or 6, characterized in that: The first device virtualization module establishes a control channel, including: In the case where the hardware device information cannot be read from the cloud server, the first device virtualization module establishes the control channel.

8. The method according to any one of claims 2 to 7, characterized in that The method further comprises: When a change is detected in the cache message queue corresponding to the first virtual device driver, the first device virtualization module reads a second request from the cache message queue, where the second request is used to request to obtain the data stream; The first terminal device receives the data stream from the second terminal device through the data channel, including: The first device virtualization module sends the data stream to the virtual bus driver based on the second request; The virtual bus driver matches and encapsulates the data stream and the second request in the cache message queue to obtain an encapsulated data packet, and feeds the encapsulated data packet back to the first application.

9. The method according to any one of claims 2 to 8, characterized in that The method further comprises: When it is detected that the second terminal device is offline, the first terminal device deletes the virtual device driver corresponding to the at least one hardware device through the virtual bus driver. The second terminal device being offline indicates that the WiFi switch and / or Bluetooth switch of at least one device among the first terminal device and the second terminal device is in the off state, or the first terminal device and the second terminal device are not logged in to the same account.

10. The method according to claim 9, characterized in that The first terminal device deletes the virtual device driver corresponding to the at least one hardware device through the virtual bus driver, including: The first communication service manager detects that the second terminal device is offline, and notifies the first device virtualization module to delete the virtual device driver corresponding to the at least one hardware device; The first device virtualization module transmits a deletion message of the virtual device driver corresponding to the at least one hardware device to the virtual bus driver; The virtual bus driver deletes the virtual device driver corresponding to the at least one hardware device based on the deletion message.

11. The method according to any one of claims 1 to 10, characterized in that The first terminal device acquires the hardware device information of the second terminal device, including: When a first switch button in the first terminal device setting interface and a second switch button in the second terminal device setting interface are in the on state, the hardware device information is obtained, the first switch button is used by the user to select whether to turn on or off the function of calling the hardware device, and the second switch button is used by the user to select whether to turn on or off the function of calling the hardware device.

12. The method according to claim 11, characterized in that The method further comprises: When the first switch button and / or the second switch button is switched from an on state to an off state, the first terminal device deletes the virtual device driver corresponding to the at least one hardware device through the virtual bus driver.

13. The method according to claim 12, characterized in that The method further comprises: The first terminal device displays a setting interface, wherein the setting interface includes the first switch button; Based on a user's selection operation on the first switch button, the first switch button is adjusted to an on state or an off state.

14. The method according to any one of claims 1 to 13, characterized in that The first terminal device includes a second application; The first terminal device installs a virtual bus driver, including: The first terminal device installs the virtual bus driver through the second application.

15. The method according to any one of claims 1 to 14, characterized in that Before the first terminal device obtains the hardware device information of the second terminal device, the method further includes: The first terminal device detects whether a calling condition of the hardware device is met, and the calling condition includes at least one of the following: The device type of the second terminal device is a preset type; The first terminal device supports the ability to call a hardware device; or, The function of the first terminal device calling the hardware device is in an enabled state; The obtaining of the hardware device information of the second terminal device includes: When the calling condition is met, the hardware device information is obtained.

16. The method according to any one of claims 1 to 15, characterized in that The first terminal device acquires the hardware device information of the second terminal device, including: The first terminal device reads the hardware device information from the cloud server.

17. The method according to any one of claims 1 to 16, characterized in that The at least one hardware device includes a camera and / or a microphone.

18. A hardware device calling device, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the device is caused to execute the method according to any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 17.

20. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 17.

Images