Cross-device hardware calling method and device

Abstract

The invention provides a cross-device hardware calling method and device, which can realize calling of a first microphone on a second terminal device by a first terminal device, so that an application program on the first terminal device can obtain audio data from the second terminal device and collected by the first microphone. The use scene of the terminal equipment is enriched, and the user experience is optimized. The method comprises the following steps: a first terminal device creates a virtual drive of a first microphone through a virtual bus drive based on a virtual drive establishment request from a second terminal device, and receives audio data collected by the first microphone from the second terminal device through calling of the virtual drive of the first microphone by a first application program.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a cross-device hardware calling method and apparatus. Background Art

[0002] For terminal devices that support sharing services, such as mobile phones, tablet computers, and laptops, users can establish a connection between a first terminal device and a second terminal device through the sharing service, so that the first terminal device and the second terminal device can control each other and data can be transmitted between the first terminal device and the first terminal device. During the process of the second terminal device controlling the first terminal device, the user can click on the interface of the first terminal device displayed on the screen of the second terminal device to view the files stored on the first terminal device.

[0003] However, currently, when the second terminal device is accessing the first terminal device, the first terminal device cannot call the microphone in the second terminal device, so in some scenarios, if the user on the second terminal device needs to speak in a meeting ongoing on the first terminal device through the microphone on the second terminal device, it is difficult to achieve, affecting the user experience. Summary of the invention

[0004] The present application provides a cross-device hardware calling method and apparatus, which establishes a virtual driver for the microphone of a second terminal device on a first terminal device, so that the first terminal device can call the first microphone on the second terminal device and receive audio data collected by the microphone of the second terminal device, which is conducive to enriching the usage scenarios of the terminal devices and optimizing the user experience.

[0005] In a first aspect, the present application provides a cross-device hardware calling method, which is applied to a system including a first terminal device and a second terminal device, a user has logged in to the same account on the first terminal device and the second terminal device, and a connection has been established between the first terminal device and the first terminal device, and the method includes: the first terminal device receives a virtual driver establishment request from the second terminal device, the virtual driver establishment request is used to request to establish a virtual driver for a first microphone on the first terminal device, the first microphone being the microphone on the second terminal device; the first terminal device creates a virtual driver for the first microphone through a virtual bus driver based on the virtual driver establishment request; the first terminal device calls the virtual driver of the first microphone through a first application, and sends a use request for the first microphone to the second terminal device; the first terminal device receives audio data collected by the second terminal device through the first microphone.

[0006] It should be understood that before the first terminal device receives the virtual driver establishment request, a virtual bus driver has been installed on the first terminal device, and the virtual bus driver is used to create and manage virtual drivers of hardware devices.

[0007] The first application may be a third-party application installed on the first terminal device, or may be a system application of the first terminal device, which is not limited in this application.

[0008] The cross-device hardware calling method provided by the present application is that, when a first terminal device and a second terminal device have established a connection, the first terminal device creates a virtual driver for the first microphone of the second terminal device on the first terminal device through a virtual bus driver based on a virtual driver establishment request received from the second terminal device. The virtual driver of the first microphone created on the first terminal device can be called by a first application on the first terminal device. Through this call, the audio data collected by the first microphone on the second terminal device is sent to the first terminal device, so that the first terminal device can use the audio data collected by the first microphone on the second terminal device, thereby realizing cross-device microphone calling. The first application on the first terminal device can use the first microphone on the second terminal device as an audio input device, enriching the usage scenarios of the terminal devices and optimizing the user experience.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the first terminal device creates a virtual driver for the first microphone through a virtual bus driver based on the virtual driver establishment request, including: a first shared service application obtains the virtual driver establishment request, the virtual driver establishment request carries device information of the first microphone, the device information of the first microphone includes device identification information of the second terminal device, identification information of the first microphone, type information of the first microphone and hardware device capability information of the first microphone; the first shared service application sends the device information of the first microphone to a first device virtualization module; the first device virtualization module creates a virtual driver for the first microphone corresponding to the attributes indicated by the device identification information, the type information and the hardware device capability information through the virtual bus driver, and establishes a correspondence between the first identification information of the virtual driver of the first microphone and the identification information of the first microphone.

[0010] It should be understood that the device identification information of the second terminal device can be any information used to indicate the second terminal device, such as the device type, device model or device ID. Exemplarily, the device identification information is the device model, such as H90pro, H90pro is used to indicate that the second terminal device is a mobile phone, and the mobile phone model is H90pro; the device identification information can also be the device name of the second terminal device set by the user, such as "Zhang San's mobile phone" and so on. The identification information of the first microphone is used to uniquely identify the hardware device of the first microphone, which can be the factory number, model, etc. of the device. The type information of the first microphone is information used to indicate that the hardware device type of the first microphone is a microphone. Different type information can correspond to different hardware devices respectively. Exemplarily, the type information of 1 can indicate that the type of the hardware device is a microphone. The hardware device capability information of the first microphone can be used to indicate the performance parameters of the first microphone, such as the sampling rate, sensitivity and signal-to-noise ratio supported by the first microphone.

[0011] In one possible implementation, the first device virtualization module creates a virtual driver for the first microphone corresponding to the attributes indicated by the device identification information, type information and hardware device capability information through a virtual bus driver. Specifically, it may include: the first device virtualization module converts the device identification information and type information into a hardware device type descriptor corresponding to the microphone in the Universal Serial Bus (USB) protocol, converts the hardware device capability information into a hardware device configuration descriptor that complies with the USB protocol and corresponds to the hardware device capability of the first microphone, and transmits the hardware device type descriptor and the hardware device configuration descriptor to the virtual bus driver, and the virtual bus driver creates a virtual driver for the first microphone corresponding to the information described by the hardware device type descriptor and the hardware device configuration descriptor.

[0012] It should be understood that the hardware device type descriptor may include information other than type information and device identification information, and the hardware device configuration descriptor may also include information other than hardware device capability information. For example, the hardware device configuration descriptor may also include interface information of the first microphone virtual driver. This application does not specifically limit the information contained in each descriptor.

[0013] It should also be understood that the first microphone virtual driver created by the virtual bus driver can be identified by the first terminal device as a microphone-type USB device, and the USB device can be used as an audio input device of the first terminal device and called by an application on the first terminal device.

[0014] In an embodiment of the present application, the virtual driver of the first microphone created by the first terminal device through the virtual bus driver can be recognized by the first terminal device as a microphone-type USB device, which facilitates the application on the first terminal device to call the device, and the device establishes a connection with the first microphone, making it possible for the first terminal device to call the first microphone on the second terminal device.

[0015] In combination with the first aspect, in certain implementations of the first aspect, before the first terminal device calls the virtual driver of the first microphone through the first application, the method also includes: the virtual bus driver sends a first notification message of successful creation of the virtual driver of the first microphone to the multimedia framework, and the first notification message includes first identification information of the virtual driver of the first microphone; the first terminal device calls the virtual driver of the first microphone through the first application, including: the multimedia framework detects that the first application is in a startup state, and sends the first identification information to the first application; the first application calls the virtual driver of the first microphone based on the first identification information.

[0016] It should be understood that the multimedia framework stores a list of hardware devices currently available on the first terminal device, and the first application needs to know the hardware devices currently available on the first terminal device through the multimedia framework. Therefore, the virtual bus driver sends the first notification message to the multimedia framework after the virtual driver of the first microphone is successfully created, so that when the multimedia framework detects that the first application is in the startup state, it notifies the first application through the multimedia framework that the virtual driver of the first microphone is available.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the first notification message also includes the creation time of the virtual driver of the first microphone. Before sending the first identification information to the first application, the method also includes: the multimedia framework adds the first identification information to the hardware device list and records the creation time; the multimedia framework determines, based on the creation time, whether the virtual driver of the first microphone is the most recently created driver in the hardware device list and the driver with the device type of microphone; sending the first identification information to the first application includes: if so, the multimedia framework sends the first identification information to the first application.

[0018] In this embodiment, the driver most recently created on the first terminal device determined by the multimedia framework is the driver that the user is most likely to want to use, so the driver is selected as the driver currently automatically called by the first application through the system default rule, which is beneficial to improving the efficiency of driver calling and enhancing the user experience.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the sending of the first identification information to the first application includes: the multimedia framework sends a hardware device list to the first application, the hardware device list includes the first identification information; the first application calls the virtual driver of the first microphone based on the first identification information, including: the first application calls the virtual driver of the first microphone in response to a user's selection operation of the first identification information in the hardware device list, the selection operation is performed by the user through an interface of the first terminal device displayed on the second terminal device, the interface of the first terminal device includes an interface of the first application, and the interface of the first application includes the hardware device list.

[0020] In this embodiment, by sending the available hardware device list to the first application, the user can select the driver he currently wants to use based on the hardware device list in the first application, avoiding the default selected driver not being the driver the user wants to select, thereby improving the user experience.

[0021] In combination with the first aspect, in some implementations of the first aspect, before the first terminal device calls the virtual driver of the first microphone through the first application, the method further includes: the first application sends a parameter negotiation request to the virtual bus driver, the parameter negotiation request is used to request the virtual bus driver to match the audio parameters corresponding to the first application for the virtual driver of the first microphone, and the parameter negotiation request carries parameter requirement information of the first application; the virtual bus driver determines the target parameters corresponding to the virtual driver of the first microphone during the transmission of the audio data based on the parameter requirement information, and sends a request success response corresponding to the parameter negotiation request to the first application; after the first terminal device receives the audio data collected by the second terminal device through the first microphone, the method further includes: the virtual bus driver processes the audio data based on the target parameters through the virtual driver of the first microphone to obtain processed audio data;

[0022] The virtual bus driver sends the processed audio data to the first application.

[0023] It should be understood that the target parameters determined by the above negotiation process are used for the virtual bus to drive the processing of the audio data after the first terminal device receives the audio data collected by the second terminal device through the first microphone, so that the processed audio data is more compatible with the first application.

[0024] Optionally, the parameter negotiated in the above negotiation process may be the sampling rate of the audio. If the sampling rate of the audio data collected by the first microphone from the second terminal device and received by the first terminal device does not match the target sampling rate, the virtual bus driver resamples the audio data through the virtual driver of the first microphone, so that the resampled audio data meets the requirements of the first application for the audio data, facilitates the transmission of the audio data by the first application, and is conducive to improving the efficiency of audio data transmission and enhancing the user experience.

[0025] In combination with the first aspect, in some implementations of the first aspect, the system also includes an application server, and the first terminal device and the second terminal device establish a connection through the application server; the first terminal device receives audio data collected from the second terminal device through the first microphone, including: the first terminal device receives the audio data collected from the second terminal device through the first microphone and forwarded by the application server through the first shared service application.

[0026] In the embodiment of the present application, the control and data transmission connection between the first terminal device and the second terminal device is based on a network connection, is not restricted by distance, and the user's usage scenarios are more extensive.

[0027] In combination with the first aspect, in certain implementations of the first aspect, the wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are both in the on state, and the first terminal device and the second terminal device establish a point-to-point P2P connection or a Bluetooth connection; the first terminal device receives audio data collected from the second terminal device through the first microphone, including: the first terminal device receives the audio data collected from the second terminal device through the first microphone through the first communication service manager.

[0028] In the embodiment of the present application, data transmission between the first terminal device and the second terminal device does not need to occupy network resources, which is conducive to saving network resources. Two terminal devices logged in to the same account can be automatically connected when they are close to each other, which is more convenient and quick.

[0029] In combination with the first aspect, in certain implementations of the first aspect, it includes: the first terminal device receives a first virtual driver deletion request from the second terminal device, the first virtual driver deletion request is used to request deletion of the virtual driver of the first microphone, and the first virtual driver deletion request includes identification information of the first microphone; the first terminal device deletes the virtual driver of the first microphone through the virtual bus driver based on the first virtual driver deletion request; the first terminal device sends a request success response corresponding to the first virtual driver deletion request to the second terminal device.

[0030] Through the method provided in this embodiment, the virtual driver of the first microphone of the first terminal device can be deleted. When the first terminal device does not need to call the first microphone on the second terminal device, timely deletion of the virtual driver can avoid the first microphone on the second terminal being occupied by the first terminal device, affecting the normal use of the second terminal device.

[0031] In combination with the first aspect, in certain implementations of the first aspect, the first terminal device deletes the virtual driver of the first microphone through the virtual bus driver based on the first virtual driver deletion request, including: a first shared service application obtains the first virtual driver deletion request; the first shared service application sends a second virtual driver deletion request to the virtual bus driver through the first device virtualization module, the second virtual driver deletion request carries first identification information of the virtual driver of the first microphone; the virtual bus driver deletes the virtual driver of the first microphone based on the first identification information.

[0032] It should be understood that when the virtual bus driver creates the virtual driver of the first microphone, it also establishes a corresponding relationship between the first identification information of the virtual driver of the first microphone and the identification information of the first microphone. When the virtual driver of the first microphone is deleted, the virtual bus driver can find the first identification information of the virtual driver of the first microphone to be deleted based on the identification information of the first microphone, thereby deleting the virtual driver of the first microphone from the first terminal device.

[0033] Through the method provided in this embodiment, after the virtual driver of the first microphone is deleted, the first application will no longer enumerate the driver, and the drivers in the hardware device list that the user can select in the first application are all available drivers, thereby preventing the virtual driver of the first microphone from being selected by the user after being deleted, which is beneficial to improving the user experience.

[0034] In a second aspect, the present application provides a cross-device hardware calling method, which is applied to a system including a first terminal device and a second terminal device, a user has logged in to the same account on the first terminal device and the second terminal device, and the first terminal device has established a connection with the first terminal device, and the method includes: the second terminal device sends a virtual driver establishment request to the first terminal device in response to a first operation of the user, the virtual driver establishment request is used to request to establish a virtual driver for a first microphone on the first terminal device, the first microphone is the microphone on the second terminal device, and the first operation is used to collect audio data for the first terminal device through the first microphone; the second terminal device receives a request to use the first microphone from the first terminal device; the second terminal device collects audio data through the first microphone based on the request to use the first microphone, and sends the audio data to the first terminal device.

[0035] In combination with the second aspect, in some implementations of the second aspect, the first operation is a user's selection operation of a preset button in a control interface on the second terminal device.

[0036] It should be understood that after the first terminal device establishes a connection with the second terminal device, an interface for managing and controlling the connection between the first terminal device and the second terminal device is set on the second terminal device. The user can trigger the terminal device to implement the function corresponding to the description of the button by operating the button set on the interface.

[0037] In combination with the second aspect, in some implementations of the second aspect, the method also includes: the second terminal device stops collecting data through the first microphone in response to a second operation of the user, and sends a first virtual drive deletion request to the first terminal device, the first virtual drive deletion request includes identification information of the first microphone, and the second operation is used to stop collecting audio data for the first terminal device through the first microphone.

[0038] In combination with the second aspect, in some implementations of the second aspect, the method further includes: the second terminal device receiving a request response corresponding to the first virtual drive deletion request from the first terminal device.

[0039] It should be understood that the request response corresponding to the first virtual drive deletion request is a successful response when the first virtual drive deletion is successful, and is a failed response when the first virtual drive deletion fails, and this application does not limit this.

[0040] In combination with the second aspect, in some implementations of the second aspect, the second operation is an operation of the user canceling a selection of the preset button in the control interface on the second terminal device.

[0041] In combination with the second aspect, in some implementations of the second aspect, the system also includes an application server, and the first terminal device and the second terminal device establish a connection through the application server; the second terminal device sends the audio data to the first terminal device, including: the second terminal device sends the audio data to the application server through a second shared service application, and the audio data carries the Internet Protocol address of the first terminal device.

[0042] In combination with the second aspect, in certain implementations of the second aspect, the wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are both in the on state, and the first terminal device and the second terminal device establish a point-to-point P2P connection or a Bluetooth connection; the second terminal device sends the audio data to the first terminal device, including: the second terminal device sends the audio data to the first terminal device through the second communication service manager.

[0043] In a third aspect, the present application provides a cross-device hardware calling system, comprising a first terminal device and a second terminal device in any possible implementation of the first aspect, respectively used to execute a method corresponding to the first terminal device and a method corresponding to the second terminal device in any possible implementation of the first aspect.

[0044] In a fourth aspect, the present application provides a cross-device hardware calling device, including: a processor and a memory, wherein the processor is used to read instructions stored in the memory to execute a method in any possible implementation of the first aspect or the second aspect above.

[0045] Optionally, there are one or more processors and one or more memories.

[0046] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0047] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. The embodiments of the present application do not limit the type of memory and the setting method of the memory and the processor.

[0048] The cross-device hardware calling device in the fourth aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor, or can be located outside the processor and exist independently.

[0049] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program (also referred to as code, or instructions). When the computer-readable storage medium is run on a computer, the computer executes a method in any possible implementation of the first aspect or the second aspect.

[0050] In a sixth aspect, the present application provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A scene graph applicable to this application;

[0052] Figure 2 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0053] Figure 3 A software structure diagram of a terminal device provided in an embodiment of the present application;

[0054] Figure 4 Another software structure block diagram of the terminal device provided in the embodiment of the present application;

[0055] Figure 5 A schematic flowchart of a cross-device hardware calling method provided in an embodiment of the present application;

[0056] Figure 6 A schematic flowchart of another cross-device hardware calling method provided in an embodiment of the present application;

[0057] Figure 7 A schematic flowchart of another cross-device hardware calling method provided in an embodiment of the present application;

[0058] Figure 8 An example diagram of a display interface of a terminal device provided in an embodiment of the present application;

[0059] Fig. 9 An example diagram of a control interface provided in an embodiment of the present application;

[0060] Fig.10 A schematic block diagram of a cross-device hardware calling apparatus provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0062] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0063] It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0064] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0065] For terminal devices that support sharing services, such as mobile phones, tablet computers, and laptops, users can establish a connection between a first terminal device and a second terminal device through the sharing service, so that the first terminal device and the first terminal device can control each other and transmit data between the first terminal device and the first terminal device. During the process of the second terminal device controlling the first terminal device, the user can click on the interface of the first terminal device displayed on the screen of the second terminal device to view the files stored on the first terminal device.

[0066] In one possible scenario, the user's first terminal device is a terminal device placed in the company office, and the user's second terminal device is a terminal device that the user carries with him. When the user is working from home or going out, he needs to attend a meeting, but due to some restrictions (the meeting is limited to terminal devices in the company's LAN, or the conference application used on the first terminal device cannot be installed on the second terminal device, etc.), the user cannot directly participate in the meeting through the second terminal device. In this scenario, the user can establish a remote access connection with the first terminal device through the second terminal device and participate in the online meeting ongoing on the first terminal device. Figure 1 In the example, the scenario 100 is illustrated by taking the first terminal device as a laptop computer and the second terminal device as a tablet computer.

[0067] Currently, in the above scenario, the user can hear the speeches of other participants because the conference application occupies or controls the speaker on the first terminal device. However, since the microphone on the second terminal device cannot be used by the conference application on the first terminal device, the voice of the user on the second terminal device cannot be transmitted to other participants through the conference application on the first terminal device, making it impossible for the user to speak directly in the meeting or participate in voice discussions, affecting the user experience.

[0068] In view of this, the present application provides a cross-device hardware calling method, which establishes a virtual driver of the first microphone on the first terminal device, so that the first microphone on the second terminal device can be called by the application on the first terminal device, and the audio data collected by the first microphone can be transmitted to the application on the first terminal device. The method provided by the present application enables the conference application in the first terminal device in the above scenario to obtain the user audio data collected by the second terminal device through the first microphone by calling the virtual driver of the first microphone, and transmit the audio data to the terminal devices corresponding to each member of the meeting, which meets the user's usage needs, is conducive to improving the user experience, and also enriches the usage scenarios of the terminal devices.

[0069] In order to better understand the embodiments of the present application, the terminal devices involved in the embodiments of the present application are introduced below. The terminal devices involved in the present application include any form of mobile phones, tablet computers, laptop computers, etc. that can implement the cross-device hardware control method provided by the present application. The present application does not specifically limit the form of the terminal devices.

[0070] Figure 2 A schematic diagram of the structure of a terminal device 200 provided in an embodiment of the present application. The first terminal device and the second terminal device in the embodiment of the present application are both applicable to Figure 2 The structure shown in the figure, but this application does not specifically limit the terminal device involved.

[0071] The terminal device 200 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0072] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 200. In other embodiments of the present application, the terminal device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0073] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0074] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0075] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0076] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0077] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 200.

[0078] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0079] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0080] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0081] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 200. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 200.

[0082] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0083] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 200, and can also be used to transmit data between the terminal device 200 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0084] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 200. In other embodiments of the present application, the terminal device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0085] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 200. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0086] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0087] The wireless communication function of the terminal device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0088] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0089] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 200. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0090] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0091] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 200. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0092] In some embodiments, the antenna 1 of the terminal device 200 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0093] The terminal device 200 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0094] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 200 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0095] The terminal device 200 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0096] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0097] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 200 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0098] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0099] Video codecs are used to compress or decompress digital videos. The terminal device 200 may support one or more video codecs. Thus, the terminal device 200 may play or record videos in multiple coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0100] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0101] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 200. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0102] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 200 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0103] The terminal device 200 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0104] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0105] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 200 can listen to music or listen to a hands-free call through the speaker 170A.

[0106] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 200 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0107] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 200 can be provided with at least one microphone 170C. In other embodiments, the terminal device 200 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 200 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0108] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0109] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 200 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 200 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 200 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0110] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 200. In some embodiments, the angular velocity of the terminal device 200 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 200 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 200 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0111] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 200 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0112] The magnetic sensor 180D includes a Hall sensor. The terminal device 200 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 200 is a flip phone, the terminal device 200 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0113] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 200 in various directions (generally three axes). When the terminal device 200 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0114] The distance sensor 180F is used to measure the distance. The terminal device 200 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 200 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0115] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 200 emits infrared light outward through the light emitting diode. The terminal device 200 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 200. When insufficient reflected light is detected, the terminal device 200 can determine that there is no object near the terminal device 200. The terminal device 200 can use the proximity light sensor 180G to detect that the user holds the terminal device 200 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0116] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 200 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 200 is in a pocket to prevent accidental touch.

[0117] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 200 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0118] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 200 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 200 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 200 heats the battery 142 to avoid abnormal shutdown of the terminal device 200 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 200 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0119] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 200, which is different from the position of the display screen 194.

[0120] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0121] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 200 may receive key input and generate key signal input related to user settings and function control of the terminal device 200.

[0122] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0123] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0124] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 200 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 200 and cannot be separated from the terminal device 200.

[0125] Combine the following Figure 3 and Figure 4 , the software structures of the first terminal device and the second terminal device involved in the cross-device hardware calling method provided in an embodiment of the present application are described in detail. Figure 3 Taking the first terminal device as a personal computer (PC) and the system of the first terminal device as a Windows system as an example, the software structure thereof is described; Figure 4 The software structure is described by taking the second terminal device being an Android tablet computer as an example.

[0126] Figure 3 is a software structure diagram of the first terminal device. The layered architecture of the Windows system divides the software into several layers, and the layers communicate with each other through software interfaces. For example, the application layer and the system layer can interact through the application programming interface (API), and the system layer and the kernel layer can interact through the standard I / O interface. In some embodiments, the Windows system is divided into three layers, namely, the application layer, the system layer, and the kernel layer from top to bottom. The specific functional modules included in each layer are explained below.

[0127] 1. Application layer

[0128] The application layer may include a series of applications, such as PC manager, Bluetooth, WIFI, third-party applications, software development kit (SDK), etc.

[0129] In an embodiment of the present application, the application layer may also include: a first shared service application, a device virtualization SDK, a communication service manager, a device virtualization module, a device manager, etc. In one possible implementation method, the first shared service application and the third-party application can access the device virtualization SDK and realize communication with the device virtualization module through the interface configured by the device virtualization SDK.

[0130] Among them, PC Manager can include installation packages of multiple applications, and users can install multiple applications for the first terminal device through PC Manager; it is also used to achieve functions such as system optimization, smart audio-visual, smart interconnection between devices, multi-screen collaboration and hotspot sharing. PC Manager can also have other names, which are not specifically limited in this application.

[0131] The shared service involved in the embodiments of the present application may also be called a smart interconnection service, a microphone (MIC) sharing service, etc., or a sub-service of smart interconnection, which is not limited in the present application.

[0132] The third-party applications mentioned above may include video conferencing, dating and chatting applications, etc.

[0133] In a possible implementation, a user can establish a connection with a second terminal device through a first shared service application on a first terminal device, and the devices that establish the connection can control the other terminal device or be controlled by the other terminal device. In an embodiment of the present application, when the second terminal device controls the first terminal device, when the application on the first terminal device calls the first microphone on the second terminal device, the audio data collected by the first microphone can be transmitted through the application server corresponding to the first shared service application, and can also be transmitted through the communication service manager.

[0134] The communication service manager may also be called a smart interconnection module, a magiclink transmission management service module, etc. This application does not make any specific limitations on this.

[0135] The device virtualization module is used to manage the life cycle of the virtual device driver, control the creation and deletion of the virtual device driver through the virtual bus driver (also known as the insertion and removal of the virtual device driver), and is also used to process requests sent by third-party applications to the virtual device driver, and feed back the processing results to the third-party application. The device virtualization module can interact with other applications through the device virtualization SDK of the application layer. In some implementations of the present application, the device virtualization module is also used to manage the data transmission channel maintained by the communication service manager to realize data transmission across devices.

[0136] The device virtualization module can also be called a hardware service cross-device sharing module, a distributed mobile sensing development platform (DMSDP), a hardware virtualization module, etc., and this application does not make specific restrictions on this. Correspondingly, the device virtualization SDK can also be called a hardware service cross-device sharing SDK, a DMSDPSDK, a hardware virtualization SDK, etc., and this application embodiment does not make any restrictions on this.

[0137] The device manager is used to view and change device properties, change device drivers, configure device settings, and uninstall devices. In the embodiment of the present application, the device manager is also used to implement security authentication and account authentication of devices with the same account, self-discovery and self-organizing networking of devices, data transmission between devices, and access to communication services.

[0138] 2. System layer

[0139] The system layer may include the Windows multimedia framework, etc., which can provide corresponding services for the execution of user application processes.

[0140] The Windows multimedia framework includes a variety of Windows function functions that can be called by applications for developing applications. The Windows multimedia framework is also used to connect the application layer and the kernel layer to achieve interaction between the application layer and the kernel layer. Exemplarily, the request sent by the application to the kernel layer can be first transmitted to the Windows multimedia framework through the Windows application framework (framwork, FWK) API, and then transmitted to the kernel layer through the standard I / O interaction interface.

[0141] 3. Kernel layer

[0142] The kernel layer is the layer between hardware and software. The kernel layer is used to drive the hardware to make it work. The kernel layer may include virtual bus drivers, virtual device drivers, Windows Driver Foundation (WDF) modules, operating system (OS) kernels, etc.

[0143] The virtual device driver can be recognized by the terminal device as a universal serial bus (USB) device and used by the terminal device. For example, a virtual camera device, a virtual microphone (mic) device. If the first terminal device recognizes an available USB device (also called a hardware device), a corresponding virtual device driver can be created for each USB device, and the number of virtual device drivers included in the kernel layer of the first terminal device can be multiple; if the first terminal device does not recognize an available USB device, the number of virtual device drivers included in the kernel layer of the terminal device 200 is 0. In the case where the first terminal device creates at least one virtual device driver, the at least one virtual device driver can be displayed in its device manager.

[0144] The virtual bus driver is used to create or delete virtual device drivers, that is, it is responsible for managing the insertion, removal and communication processes of virtual device drivers.

[0145] Optionally, PC Manager can call the installation script and installation package of the virtual bus driver to install the virtual bus driver.

[0146] The virtual bus driver may also be referred to as a virtual bus controller, a virtual driver bus, etc., and this application does not make any specific limitation on this.

[0147] The OS kernel is used to convert requests from the application layer into data processing instructions and send them to the virtual device driver through the driver development framework.

[0148] The driver development framework is used to implement sending data processing instructions from the OS kernel to the virtual device driver.

[0149] Figure 4 A software structure diagram of a second terminal device provided in an embodiment of the present application. The Android layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, the hardware abstraction layer, and the Linux kernel layer.

[0150] 1. Application layer

[0151] The application layer can include a series of application packages. Figure 4 As shown, in the embodiment of the present application, the application layer includes at least application packages such as a third-party application, a second shared service application, Bluetooth, WIFI, and a device virtualization SDK.

[0152] The second shared service application can be an independent application or integrated into an application with the device virtualization SDK. The functions corresponding to the second shared service application include remote control services, data sharing services, etc. In the embodiment of the present application, the user can find the service in the smart interconnection interface by clicking Settings > Smart Interconnection in sequence.

[0153] The third-party application may be an application such as a video conference application or a dating chat application.

[0154] Among them, the device virtualization module SDK can also include public service interface, camera service interface, audio service interface and speaker service interface, etc. The interfaces of these services are exposed through the Android interface definition language (AIDL) provided by the interface object abstraction layer. The interface object abstraction layer can be encapsulated by the service interface adaptation layer to achieve cross-process access.

[0155] In an embodiment of the present application, the device virtualization module SDK can call the device virtualization module in the system library through a public service interface.

[0156] 2. Application framework layer

[0157] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0158] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes at least: a communication service manager, a device manager, an audio framework, etc.

[0159] The communication service manager may also be called a smart interconnection module, a magiclink transmission management service module, etc. This application does not make any specific limitations on this.

[0160] The communication service manager of the above-mentioned first terminal device can be called the first communication service manager, the communication service manager of the second terminal device can be called the second communication service manager, and the data transmission channel between the first communication service manager and the second communication service manager can be called the magic link data transmission channel, which is not limited in this application.

[0161] 3. System layer

[0162] The system layer includes the system library and the Android runtime. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0163] The system library may include multiple functional modules. In one possible implementation, the system library includes at least an audio processing module for processing and transmitting audio data collected by a microphone; in another possible implementation, the system library may also include: a device virtualization module, which may be responsible for interacting with hardware devices, such as turning on and off the microphone, obtaining audio data collected by the microphone, encoding and decoding audio data, and may also process messages from the second shared service application through interaction with the device virtualization module SDK.

[0164] The device virtualization module may also include an interface security management module, a distribution control module, a channel management module, a device management module, a service management module, etc. Among them, in the embodiment of the present application, the service management module may include a microphone service module, and the microphone service module may also include an audio data processing module and a codec module. Optionally, the service management module may also include a camera service module and a speaker service module, and the present application does not limit the hardware service modules and functions contained therein.

[0165] 4. Hardware Abstraction Layer

[0166] The hardware abstraction layer is an abstracted structure between the Linux kernel layer and the system runtime layer. The hardware abstraction layer can be a package of the hardware driver, providing a unified interface for the call of the upper layer application. The hardware abstraction layer may include a Bluetooth module, a WIFI module, a hardware configuration module, etc. For example, in the embodiment of the present application, the Bluetooth module and the WIFI module in the hardware abstraction layer can shield the specific hardware implementation details of the Bluetooth driver and the WIFI driver in the kernel layer, and the upper layer application can implement the Bluetooth function and the WIFI function by calling the interface without having to know the specific implementation method of the kernel layer hardware driver.

[0167] 5. Linux kernel layer

[0168] The Linux kernel layer is the layer between hardware and software. The Linux kernel layer contains at least display drivers, audio drivers, memory drivers, sensor drivers, inter-process communication drivers, etc.

[0169] Combine the following Figures 5 to 9 The cross-device hardware calling method provided by the present application is described in detail. The specific form and quantity of each device shown in the embodiment shown in the present application are only examples and should not constitute any limitation on the implementation of the method provided by the present application. Below, taking the first terminal device and the second terminal device as the execution subject as an example, the cross-device hardware calling method of the embodiment of the present application is described in detail.

[0170] It should be understood that the terminal device can be the terminal device itself, or it can be a chip, chip system or processor that supports the terminal device to implement a cross-device hardware calling method, or it can be a logic module or software that can implement all or part of the terminal device functions. This application does not make specific limitations on this.

[0171] Figure 5 A schematic flowchart of a cross-device hardware calling method 500 provided in an embodiment of the present application. The method 500 is applied to a system including a first terminal device and a second terminal device, a user logs in to the same account on the first terminal device and the second terminal device, and the first terminal device and the second terminal device have established a connection.

[0172] It should be understood that an account may refer to a personal registered account provided by different terminal device brands to ensure the security of the user's device, such as a Honor account. The first terminal device and the second terminal device logged in to the same account are considered to be mutually trustworthy devices, and can establish a connection through a remote or near-field network, a peer-to-peer (P2P) connection, or Bluetooth. After the connection is established, communication can be achieved between the first terminal device and the second terminal device. This application does not limit the connection method between the first terminal device and the second terminal device.

[0173] The method 500 specifically includes the following steps:

[0174] S501. The second terminal device responds to a first operation of a user. The first operation is used to collect audio data for the first terminal device through a first microphone. The first microphone is a microphone on the second terminal device.

[0175] S502: The second terminal device sends a virtual driver establishment request to the first terminal device, where the virtual driver establishment request is used to request to establish a virtual driver of the first microphone on the first terminal device; correspondingly, the first terminal device receives the virtual driver establishment request.

[0176] S503: The first terminal device creates a virtual driver for the first microphone through a virtual bus driver based on the virtual driver establishment request.

[0177] S504: The first terminal device calls a virtual driver of the first microphone through the first application.

[0178] S505. The first terminal device sends a request to use the first microphone to the second terminal device; correspondingly, the second terminal device receives the request to use the first microphone.

[0179] S506: The second terminal device collects audio data through the first microphone based on the request to use the first microphone.

[0180] S507. The second terminal device sends the audio data collected by the first microphone to the first terminal device; correspondingly, the first terminal device receives the audio data.

[0181] It should be understood that after the first terminal device establishes a connection with the second terminal device, an interface for managing and controlling the connection between the first terminal device and the second terminal device is set on the first terminal device and / or the second terminal device, and the user can trigger the terminal device to implement the function corresponding to the description of the button by operating the button set on the interface.

[0182] It should also be understood that before S502 , a virtual bus driver has been installed on the first terminal device, and the virtual bus driver is used to create and manage virtual drivers of hardware devices.

[0183] The first application may be a third-party application installed on the first terminal device, or may be a system application of the first terminal device, which is not limited in this application.

[0184] The cross-device hardware calling method provided by the present application is that, when a first terminal device and a second terminal device have established a connection, the first terminal device creates a virtual driver for the first microphone of the second terminal device on the first terminal device through a virtual bus driver based on a virtual driver establishment request received from the second terminal device. The virtual driver of the first microphone created on the first terminal device can be called by a first application on the first terminal device. Through this call, the audio data collected by the first microphone on the second terminal device is sent to the first terminal device, so that the first terminal device can use the audio data collected by the first microphone on the second terminal device, thereby realizing cross-device microphone calling. The first application on the first terminal device can use the first microphone on the second terminal device as an audio input device, enriching the usage scenarios of the terminal devices and optimizing the user experience.

[0185] As an optional embodiment, the first operation in the above S501 is a user selecting a preset button in a control interface on the second terminal device, and the control interface may be an interface corresponding to the second shared service application on the second terminal device. The user may select the preset button in any possible manner such as single-clicking, double-clicking, sliding, or knuckle tapping with the fingertips, and this application does not limit this. Exemplarily, the name of the preset button may be "input sound on this device", or any similar expression.

[0186] After the user logs in to the same account on the first terminal device and the second terminal device, the first terminal device and the second terminal device can establish a connection in the following two ways.

[0187] Method 1: The first terminal device and the second terminal device are both connected to the network, and a remote access network connection is established between the second terminal device and the first terminal device through the first shared service application on the first terminal device and the second shared service application on the second terminal device. During access, the user can see the content on the interface of the first terminal device through the interface of the second terminal device, and can also operate the first terminal device through the screen of the second terminal device or the keyboard, mouse and other devices connected to the second terminal device.

[0188] In this connection mode, the first shared business application and the second shared business application can be applications used to implement remote access connection between the first terminal device and the second terminal device. In one possible implementation method, the shared business application can communicate with the device virtualization module through the interface provided by the device virtualization SDK; in another possible implementation method, the functions corresponding to the shared business application can be integrated into the device virtualization SDK, and this application does not make specific limitations on this.

[0189] It should also be understood that the so-called "remote access of the second terminal device to the first terminal device" means that the user can operate the first terminal device through the second terminal device, and is not a specific limitation on the physical distance between the first terminal device and the second terminal device.

[0190] Method 2: When the WiFi switches and Bluetooth switches of the first terminal device and the second terminal device are both turned on, the first terminal device can detect the second terminal device (the distance between the first terminal device and the second terminal device is within a preset range). Since the first terminal device and the second terminal device are logged in to the same account, the first terminal device and the second terminal device establish a point-to-point P2P connection or Bluetooth connection.

[0191] In this connection mode, the first shared service application and the second shared service application can be applications for realizing remote access connection between the first terminal device and the second terminal device, or applications for realizing data sharing between the first terminal device and the second terminal device. That is, in this connection mode, the user can choose whether to establish a connection in the form of remote access or in the form of data sharing only. The difference is that if the second terminal device remotely accesses the first terminal device, the user can see the interface displayed on the first terminal device through the screen of the second terminal device and can also operate the interface. If it is a data sharing connection, the two devices can communicate and transmit data, but the user cannot see the display interface of the other terminal on the first terminal device or the second terminal device, nor can he operate the interface of the other terminal device.

[0192] It is worth noting that if the first terminal device and the second terminal device establish a connection in different ways, the implementation method of the above S507 will also be different.

[0193] As an optional embodiment, when the first terminal device and the second terminal device establish a connection in the above-mentioned manner, the system in which the first terminal device and the second terminal device are located further includes an application server, and the above-mentioned S507 is implemented as follows: the second terminal device sends audio data collected by the first microphone to the application server through the second shared service application, and the audio data carries the Internet protocol (IP) address of the first terminal device, and the application server sends the audio data to the first terminal device based on the IP address; correspondingly, the first terminal device receives the audio data from the second terminal device and forwarded by the application server through the first shared service application.

[0194] In the embodiment of the present application, the connection between the first terminal device and the second terminal device is not restricted by distance, and the user's usage scenarios are more extensive.

[0195] As an optional embodiment, when the first terminal device and the second terminal device establish a connection in the above-mentioned method 2, the implementation method of the above-mentioned S507 is: the second terminal device sends audio data to the first terminal device through the second communication service manager; correspondingly, the first terminal device receives audio data collected from the second terminal device through the first microphone through the first communication service manager.

[0196] It should be understood that when the first terminal device and the second terminal device establish a connection in the second manner described above, a data transmission channel is established between the first communication service manager and the second communication service manager on the first terminal device to implement data transmission between the two devices.

[0197] In the embodiment of the present application, data transmission between the first terminal device and the second terminal device does not need to occupy network resources, which is conducive to saving network resources.

[0198] As an optional embodiment, a possible implementation method of the above S503 includes: the first shared service application obtains a virtual driver establishment request, the virtual driver establishment request carries the device information of the first microphone, the device information of the first microphone includes the device identification information of the second terminal device, the identification information of the first microphone, the type information of the first microphone and the hardware device capability information of the first microphone; the first shared service application sends the device information of the first microphone to the first device virtualization module; the first device virtualization module creates a virtual driver of the first microphone corresponding to the attributes indicated by the device identification information, the type information and the hardware device capability information through a virtual bus driver, and establishes a correspondence between the first identification information of the virtual driver of the first microphone and the identification information of the first microphone.

[0199] It should be understood that the device identification information of the second terminal device can be any information used to indicate the second terminal device, such as the device type, device model or device ID. Exemplarily, the device identification information is the device model, such as H90pro, H90pro is used to indicate that the second terminal device is a mobile phone, and the mobile phone model is H90pro; the device identification information can also be the device name of the second terminal device set by the user, such as "Zhang San's mobile phone" and so on. The identification information of the first microphone is used to uniquely identify the hardware device of the first microphone, which can be the factory number, model, etc. of the device. The type information of the first microphone is information used to indicate that the hardware device type of the first microphone is a microphone. Different type information can correspond to different hardware devices respectively. Exemplarily, the type information of 1 can indicate that the type of the hardware device is a microphone. The hardware device capability information of the first microphone can be used to indicate the performance parameters of the first microphone, such as the sampling rate, sensitivity and signal-to-noise ratio supported by the first microphone.

[0200] In one possible implementation, the first device virtualization module creates a virtual driver for the first microphone corresponding to the attributes indicated by the device identification information, type information and hardware device capability information through a virtual bus driver. Specifically, it may include: the first device virtualization module converts the device identification information and type information into a hardware device type descriptor corresponding to the microphone in the Universal Serial Bus (USB) protocol, converts the hardware device capability information into a hardware device configuration descriptor that complies with the USB protocol and corresponds to the hardware device capability of the first microphone, and transmits the hardware device type descriptor and the hardware device configuration descriptor to the virtual bus driver, and the virtual bus driver creates a virtual driver for the first microphone corresponding to the information described by the hardware device type descriptor and the hardware device configuration descriptor.

[0201] It should be understood that the hardware device type descriptor may include information other than type information and device identification information, and the hardware device configuration descriptor may also include information other than hardware device capability information. For example, the hardware device configuration descriptor may also include interface information of the first microphone virtual driver. This application does not specifically limit the information contained in each descriptor.

[0202] It should also be understood that the first microphone virtual driver created by the virtual bus driver can be identified by the first terminal device as a microphone-type USB device, and the USB device can be used as an audio input device of the first terminal device and called by an application on the first terminal device.

[0203] In an embodiment of the present application, the virtual driver of the first microphone created by the first terminal device through the virtual bus driver can be recognized by the first terminal device as a microphone-type USB device, which facilitates the application on the first terminal device to call the device, and the device establishes a connection with the first microphone, making it possible for the first terminal device to call the first microphone on the second terminal device.

[0204] As an optional embodiment, before the first terminal device executes S504, the virtual bus driver of the kernel layer on the first terminal device sends a first notification message of successful creation of the virtual driver of the first microphone to the multimedia framework of the system layer, and the first notification message includes first identification information of the virtual driver of the first microphone; a possible implementation method of the above S504 includes: the multimedia framework detects that the first application is in the startup state, and sends the first identification information to the first application; the first application calls the virtual driver of the first microphone based on the first identification information.

[0205] It should be understood that the multimedia framework stores a list of hardware devices currently available on the first terminal device, and the first application needs to know the hardware devices currently available on the first terminal device through the multimedia framework. Therefore, the virtual bus driver sends the first notification message to the multimedia framework after the virtual driver of the first microphone is successfully created, so that when the multimedia framework detects that the first application is in the startup state, it notifies the first application through the multimedia framework that the virtual driver of the first microphone is available.

[0206] In a possible implementation, the driver called by the first application is selected by the multimedia framework based on the system default rules. When the multimedia framework detects that the first application is in the startup state, it sends the most recently created driver in the hardware device list whose device type is a microphone to the first application. Whether the driver is the most recently created can be determined by the creation time of each driver stored in the list. If it is determined that the virtual driver of the first microphone is the most recently created driver, the multimedia framework sends the first identification information corresponding to the virtual driver of the first microphone to the first application, and then the first application calls the virtual driver of the first microphone based on the first identification information. Optionally, the identification information may include interface information of the virtual driver of the first microphone.

[0207] It can be understood that the driver most recently created on the first terminal device determined by the multimedia framework is the driver that the user is most likely to want to use, so the driver is selected as the driver currently automatically called by the first application through the system default rule, which is conducive to improving the efficiency of driver calling and enhancing the user experience.

[0208] In another possible implementation, the driver called by the first application is selected by the user based on the hardware device list. Before implementing this call, the multimedia framework sends a hardware device list to the first application, the list includes first representation information, and the first application calls the virtual driver of the first microphone in response to the user's selection operation on the first identification information. The selection operation can be performed by the user through the interface of the first terminal device displayed on the second terminal device, the interface of the first terminal device includes the interface of the first application, and the interface of the first application includes the hardware device list.

[0209] In this embodiment, by sending the available hardware device list to the first application, the user can select the driver he currently wants to use based on the hardware device list in the first application, avoiding the default selected driver not being the driver the user wants to select, thereby improving the user experience.

[0210] As an optional embodiment, before S504, the first application on the first terminal device can also determine the target parameters corresponding to the virtual driver of the first microphone during audio data transmission through parameter negotiation with the virtual bus driver. The parameter negotiation process specifically includes: the first application sends a parameter negotiation request to the virtual bus driver, the parameter negotiation request is used to request the virtual bus driver to match the audio parameters corresponding to the first application for the virtual driver of the first microphone, and the parameter negotiation request carries the parameter requirement information of the first application; the virtual bus driver determines the target parameters corresponding to the virtual driver of the first microphone during audio data transmission based on the parameter requirement information, and sends a request success response corresponding to the parameter negotiation request to the first application.

[0211] It should be understood that the target parameters determined by the above negotiation process are used for the virtual bus to drive the processing of the audio data after the first terminal device receives the audio data collected by the second terminal device through the first microphone, so that the processed audio data is more compatible with the first application.

[0212] Exemplarily, the parameter negotiated in the above negotiation process may be the sampling rate of the audio. If the sampling rate of the audio data collected by the first microphone from the second terminal device and received by the first terminal device does not match the target sampling rate, the virtual bus driver resamples the audio data through the virtual driver of the first microphone, so that the resampled audio data meets the requirements of the first application for the audio data, facilitates the transmission of the audio data by the first application, and is conducive to improving the efficiency of audio data transmission and enhancing the user experience.

[0213] Figure 6 The following is a schematic flow chart of a cross-device hardware calling method 600 provided in an embodiment of the present application. The method 600 is applied to a system including a first terminal device and a second terminal device. The method 600 is executed after the above S503 and specifically includes the following steps:

[0214] S601. The second terminal device stops collecting data through the first microphone in response to a second operation of the user, where the second operation is used to stop collecting audio data for the first terminal device through the first microphone.

[0215] S602: The second terminal device sends a first virtual driver deletion request to the first terminal device, where the first virtual driver deletion request includes identification information of the first microphone; correspondingly, the first terminal device receives the first virtual driver deletion request.

[0216] S603: The first terminal device deletes the virtual driver of the first microphone through the virtual bus driver based on the first virtual driver deletion request.

[0217] S604: The first terminal device sends a request success response corresponding to the first virtual drive deletion request to the second terminal device; correspondingly, the second terminal device receives the request success response.

[0218] It should be understood that, when the execution result of S603 is that the virtual driver deletion of the first microphone fails, what the first terminal device sends to the second terminal device in S604 is a request failure response corresponding to the first virtual driver deletion request.

[0219] As an optional embodiment, the second operation after the above-mentioned first operation is a deselection operation of the user on a selected preset button in the control interface on the second terminal device.

[0220] Through the method provided in this embodiment, the virtual driver of the first microphone of the first terminal device can be deleted. When the first terminal device does not need to call the first microphone on the second terminal device, timely deletion of the virtual driver can avoid the first microphone on the second terminal being occupied by the first terminal device, affecting the normal use of the second terminal device.

[0221] As an optional embodiment, a possible implementation method of the above S603 includes: the first shared business application obtains a first virtual driver deletion request; the first shared business application sends a second virtual driver deletion request to the virtual bus driver through the first device virtualization module, and the second virtual driver deletion request carries the first identification information of the virtual driver of the first microphone; the virtual bus driver deletes the virtual driver of the first microphone based on the first identification information.

[0222] It should be understood that when the virtual bus driver creates the virtual driver of the first microphone in the above method 500, a corresponding relationship between the first identification information of the virtual driver of the first microphone and the identification information of the first microphone is also established. When the virtual driver of the first microphone is deleted, the virtual bus driver can find the first identification information of the virtual driver of the first microphone to be deleted based on the identification information of the first microphone, thereby deleting the virtual driver of the first microphone from the first terminal device.

[0223] In one possible implementation, in order to avoid the first microphone's virtual driver still being displayed as available in the hardware device list of the first application after the driver is deleted, thereby affecting the user experience, in an embodiment of the present application, after the virtual bus deletes the virtual driver of the first microphone, a notification message that the virtual driver of the first microphone has been deleted is sent to the multimedia framework. After receiving the message, the multimedia framework notifies the first application to update the hardware device list.

[0224] Through the method provided in this embodiment, after the virtual driver of the first microphone is deleted, the first application will no longer enumerate the driver, and the drivers in the hardware device list that the user can select in the first application are all available drivers, thereby preventing the virtual driver of the first microphone from being selected by the user after being deleted, which is beneficial to improving the user experience.

[0225] Combine the following Figure 7 Taking the first terminal device as a laptop computer with Windows system and the second terminal device as a tablet computer with Android system as an example, the cross-device hardware calling method provided by this application is described in detail.

[0226] Figure 7A schematic flowchart of a cross-device hardware calling method 700 provided in an embodiment of the present application, in which a user logs into the same account on a first terminal device and a second terminal device, and a connection is established between the first terminal device and the second terminal device.

[0227] The hardware structure of the first terminal device can be as follows Figure 2 As shown, the software structure can be as follows Figure 3 As shown. The first terminal device at least includes a first shared service application, a device virtualization module, a virtual bus driver and a first application. Optionally, the first application is a conference application.

[0228] The hardware structure of the second terminal device can be as follows Figure 2 As shown, the software structure can be as follows Figure 4 The second terminal device at least includes a first shared service application, and the user can establish a connection with the first terminal device through the second shared service application of the second terminal device.

[0229] The method 700 specifically includes the following steps:

[0230] S701. A second shared service application program responds to a first operation of a user selecting a preset button on a control interface. The first operation is used to collect audio data for a first terminal device through a first microphone, where the first microphone is a microphone on a second terminal device.

[0231] S702: The second shared service application sends a virtual driver establishment request to the first shared service application, where the virtual driver establishment request carries device information of the first microphone; correspondingly, the first shared service application receives the virtual driver establishment request.

[0232] S703: The first shared service application enables a first device virtualization module and carries device information of the first microphone.

[0233] S704: The first device virtualization module converts the device information of the first microphone into a descriptor that complies with the USB protocol and corresponds to the device information of the first microphone.

[0234] Specifically, the device information of the first microphone includes the device identification information of the second terminal device, the identification information of the first microphone, the type information of the first microphone and the hardware device capability information of the first microphone. The first device virtualization module converts the device identification information and the type information into a hardware device type descriptor corresponding to the microphone in the Universal Serial Bus (USB) protocol, and converts the hardware device capability information into a hardware device configuration descriptor that complies with the USB protocol and corresponds to the hardware device capability of the first microphone.

[0235] S705. The first device virtualization module sends the descriptor to the virtual bus driver; correspondingly, the virtual bus driver receives the descriptor.

[0236] S706: The virtual bus driver creates a virtual bus driver corresponding to the descriptor.

[0237] Specifically, the virtual bus driver creates a virtual driver of the first microphone corresponding to the information described by the hardware device type descriptor and the hardware device configuration descriptor, and establishes a correspondence between first identification information of the virtual driver of the first microphone and identification information of the first microphone.

[0238] S707. The virtual bus driver sends a first notification message to the Windows Multimedia Framework indicating that the virtual driver of the first microphone is successfully created. The first notification message carries first identification information of the virtual driver of the first microphone. Correspondingly, the Windows Multimedia Framework receives the first notification message.

[0239] Optionally, after receiving the first notification message, the Windows multimedia framework sends a USB protocol confirmation request to the virtual bus driver, the request being used to confirm whether the USB protocol corresponding to the virtual driver of the first microphone is a protocol that the Windows multimedia framework can parse. Correspondingly, the virtual bus driver returns a confirmation response to the Windows multimedia framework.

[0240] S708: The Windows multimedia framework adds the first identification information of the virtual driver of the first microphone to a hardware device list.

[0241] Optionally, after the virtual driver of the first microphone is successfully created, the Windows multimedia framework may return a virtual driver creation success response to the second service application through the first device virtualization module and the first shared service application.

[0242] The Windows Multimedia framework detects the first application (in Figure 1 In the scenario, when the first application (that is, the conference application used by the user) is in the startup state, S709 is continued to be executed.

[0243] S709. The Windows multimedia framework sends the hardware device list to the first application; correspondingly, the first application receives the hardware device list.

[0244] S710: The first application program enumerates a virtual driver of the first microphone based on the hardware device list.

[0245] S711. The first application sends a virtual driver parameter negotiation request of the first microphone to the virtual bus driver, carrying parameter requirement information of the first application. Correspondingly, the virtual bus driver receives the parameter negotiation request.

[0246] S712: The virtual bus driver determines, based on the parameter negotiation request, target parameters corresponding to the virtual driver of the first microphone during audio data transmission.

[0247] S713: The virtual bus driver returns a successful response to the parameter negotiation request to the first application; correspondingly, the first application receives the response.

[0248] If the first application does not use any audio input device when the virtual driver of the first microphone is created, then when the first application enumerates the virtual driver of the first microphone, the virtual driver of the first microphone is selected as the current audio input device by default; if the first application is using the second microphone of the first terminal device as the audio input device when the virtual driver of the first microphone is created, the user can select the virtual driver of the first microphone as the current audio input device from the hardware device list through the interface of the first application on the first terminal device displayed on the second terminal device.

[0249] like Figure 8 As shown, taking the second terminal device as a tablet computer as an example, the interface 801 of the first terminal device (laptop computer) is displayed on the tablet computer. The user can display the "Select Microphone" list 802 by clicking the expansion symbol on the right side of the "Mute" icon, and the name of the virtual driver of the first microphone can be displayed as "The first microphone of Zhang San's PAD", where "Zhang San's PAD" is the communication name edited by the user for the second terminal device, and "The first microphone" indicates the microphone of the second terminal device. The user can select the virtual driver of the first microphone as the current audio input device by clicking the corresponding area of ​​"The first microphone of Zhang San's PAD" on the second terminal device.

[0250] It should be understood that the name of the virtual driver of the first microphone may change as the user edits the broadcast name of the second terminal device, or may be named after the model of the second terminal device, which is not limited in the present application.

[0251] It should also be understood that when the user is using the first application, if he does not need to collect audio data through the first microphone temporarily, he can use Figure 8 The list 802 shown selects other microphones as audio input devices. When the user switches to other microphones, the virtual driver of the first microphone still exists, and the user can still switch the audio input device to the virtual driver of the first microphone by selecting from the list.

[0252] When the virtual driver of the first microphone is selected as the current audio input device, S714 is executed.

[0253] S714: The first application sends a request to use the first microphone to the virtual bus driver.

[0254] Correspondingly, the virtual bus driver receives the request and forwards the request to the first shared service application through the first device virtualization module, and the first shared service application sends the request to use the first microphone to the second shared service application.

[0255] S715: The second shared service application collects audio data through the first microphone.

[0256] S716: The second shared service application sends the audio data to the first terminal device.

[0257] S717: The first device virtualization module puts the received audio data into a data cache queue.

[0258] S718: The first application sends a data request to the virtual bus driver.

[0259] Correspondingly, the virtual bus driver receives the data request and forwards the data request to the first device virtualization module through the virtual driver of the first microphone.

[0260] After the first device virtualization module receives the data request, S719 is executed.

[0261] S719: The first device virtualization module matches and packages the audio data in the data cache queue based on the data request to obtain first audio data corresponding to the data request.

[0262] It should be understood that the data request includes the data volume and data format requested this time. The first device virtualization module obtains the first audio data of the corresponding data volume from the data cache queue, and encapsulates it in the data format required by the data request to obtain a data packet corresponding to the data format, and sends the data packet to the virtual bus driver.

[0263] S720. The first device virtualization module sends the first audio data to the virtual bus driver. Correspondingly, the virtual bus driver receives the first audio data.

[0264] S721: The virtual bus driver processes the first audio data based on the target parameter through the first microphone virtual driver to obtain processed first audio data.

[0265] It should be understood that the target parameter is the parameter determined in S712.

[0266] S722: The virtual bus driver sends the processed first audio data to the first application program through the first microphone virtual driver.

[0267] S723: The second sharing service application stops collecting audio data for the first terminal device through the first microphone in response to the user's second operation of selecting a preset button on the control interface.

[0268] S724: The second sharing service application sends a first virtual driver deletion request to the first sharing service application, where the request carries identification information of the first microphone; correspondingly, the first sharing service application receives the request.

[0269] S725. The first shared service application program disables the first device virtualization module.

[0270] S726. The first device virtualization module sends a second virtual driver deletion request to the virtual bus driver, where the request carries the first identification information of the virtual driver of the first microphone; correspondingly, the virtual bus driver receives the request.

[0271] S727: The virtual bus driver deletes the virtual driver of the first microphone.

[0272] It can be understood that a USB device with a device type of microphone is unplugged from the virtual bus driver.

[0273] S728. The virtual bus driver sends a first notification message to the Windows Multimedia Framework that the virtual driver of the first microphone has been deleted; correspondingly, the Windows Multimedia Framework receives the notification.

[0274] S729, Windows Multimedia Framework updates the hardware device list.

[0275] It should be understood that the first identification information of the virtual driver of the first microphone in the updated hardware device list has been deleted.

[0276] S730. The Windows Multimedia Framework sends a list update notification to the first application.

[0277] Correspondingly, the first application updates the hardware device list.

[0278] S731. The first application sends an update success response to the Windows Multimedia Framework.

[0279] S732. The Windows multimedia framework sends a second notification response to the virtual bus driver.

[0280] S733: The virtual bus driver sends a request success response corresponding to the second virtual driver deletion request to the first device virtualization module.

[0281] S734. The first device virtualization module returns an enabling success response to the first shared service application.

[0282] S735: The first shared service application sends a request success response corresponding to the first virtual drive deletion request to the second shared service application.

[0283] It should be understood that after the virtual driver of the first microphone is successfully deleted, if the user does not disconnect the remote control connection of the second terminal device to the first terminal device, the user can still remotely access the first terminal device through the second terminal device.

[0284] In a possible implementation, the connection mode of the first terminal device and the second terminal device is the above-mentioned mode 1, and the systems of the first terminal device and the second terminal device also include an application server ( Figure 7 (not shown), the above S716 specifically includes: the second shared service application sends audio data to the application server, and carries the IP address of the first terminal device. After receiving the audio data, the application server sends the audio data to the first terminal device based on the IP address of the first terminal device. Correspondingly, the first terminal device receives the audio data through the first shared service application. After the first shared service application decodes the received audio data, it sends the audio data to the first device virtualization module. Exemplarily, this audio data transmission method can be called sound network (a cloud for transmitting sound) transmission, which is not limited in this application.

[0285] In another possible implementation, the first terminal device and the second terminal device are connected in a P2P or Bluetooth manner as described in the above manner 2, and the first terminal device further includes a first communication service manager ( Figure 7 ), the second terminal device also includes a second communication service manager ( Figure 7 The above S716 specifically includes: the audio data collected by the first microphone is transmitted to the first communication service manager through the second communication service manager, and then transmitted to the data cache queue of the first device virtualization module by the first communication service manager, and the first device virtualization module decodes the audio data.

[0286] It should be understood that the order of execution of the above methods does not necessarily mean the order in which they are executed. The order in which the methods are executed should be determined by their functions and internal logic.

[0287] Fig. 9An example diagram of the control interface on the terminal device is given as an example. The control interface includes operation buttons such as locking PC, disconnecting, HD mode, matching tablet resolution, playing sound on secondary device, and inputting sound on this device. If the user selects "lock PC", the first terminal device can only be operated through the second terminal device, and the operation on the first terminal device will not be responded to by the local device; if the user selects "disconnect", the remote connection of the second terminal device to the first terminal device will be disconnected, and the user will not be able to operate the first terminal device through the second terminal device; the user can also pop up a submenu when selecting "HD mode" for the user to select the image quality of the first terminal device interface displayed on the screen of the second terminal device; after selecting "match tablet definition", the interface on the first terminal device can be adapted to the definition parameters corresponding to the tablet and displayed; select "play sound on this device" to play the sound played by the speaker of the first terminal device through the second terminal device.

[0288] For example, in the above methods 500 and 700, the user's first operation may be to Fig. 9 The selection operation of the "Input sound on this device" button shown in ; in method 700 and method 700, the user's second operation may be a deselection operation of the "Input sound on this device" button, but the present application is not limited to this.

[0289] Combination of the above Figures 2 to 9 , describes in detail the cross-device hardware calling method provided by this application, and the following is combined with Fig.10 , describes in detail the cross-device hardware calling device provided by this application.

[0290] Fig.10 The embodiment of the present application provides a cross-device hardware calling device 1000. The cross-device hardware calling device 1000 includes a processor 1001, a transceiver 1002, and a memory 1003. The processor 1001, the transceiver 1002, and the memory 1003 communicate with each other through an internal connection path, the memory 1003 is used to store instructions, and the processor 1001 is used to execute the instructions stored in the memory 1003 to control the transceiver 1002 to send signals and / or receive signals.

[0291] It should be understood that the cross-device hardware calling device 1000 can be specifically the first terminal device or the second terminal device in the above embodiment, and can be used to execute the various steps and / or processes corresponding to the first terminal device or the second terminal device in the above method embodiment. Optionally, the memory 1003 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. The processor 1001 may be used to execute instructions stored in the memory, and when the processor 1001 executes the instructions stored in the memory, the processor 1001 is used to execute the various steps and / or processes of the above method embodiment corresponding to the first terminal device or the second terminal device. The transceiver 1002 may include a transmitter and a receiver, the transmitter may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the sending action, and the receiver may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the receiving action.

[0292] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0293] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0294] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method described in the above embodiment.

[0295] The present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method described in the above embodiment.

[0296] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0297] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0298] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0299] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0300] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0301] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0302] The above are only specific implementation methods of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be based on the protection scope of the claims.

Claims

1. A cross-device hardware calling method, characterized in that: Applied to a system including a first terminal device and a second terminal device, a user logs in to the same account on the first terminal device and the second terminal device, and the first terminal device and the second terminal device have established a connection, the method comprising: The first terminal device receives a virtual driver establishment request from the second terminal device, wherein the virtual driver establishment request is used to request to establish a virtual driver of a first microphone on the first terminal device, and the first microphone is a microphone on the second terminal device; The first terminal device creates a virtual driver for the first microphone through a virtual bus driver based on the virtual driver establishment request; The first terminal device calls a virtual driver of the first microphone through a first application program, and sends a request for using the first microphone to the second terminal device; The first terminal device receives audio data collected by the second terminal device through the first microphone.

2. The method according to claim 1, characterized in that The first terminal device creates a virtual driver for the first microphone through a virtual bus driver based on the virtual driver establishment request, including: The first shared service application obtains the virtual driver establishment request, where the virtual driver establishment request carries device information of the first microphone, where the device information of the first microphone includes device identification information of the second terminal device, identification information of the first microphone, type information of the first microphone, and hardware device capability information of the first microphone; The first shared service application sends the device information of the first microphone to the first device virtualization module; The first device virtualization module creates a virtual driver of the first microphone corresponding to the device identification information, the type information, and the attributes indicated by the hardware device capability information through the virtual bus driver, and establishes a corresponding relationship between the first identification information of the virtual driver of the first microphone and the identification information of the first microphone.

3. The method according to claim 1 or 2, characterized in that: Before the first terminal device calls the virtual driver of the first microphone through the first application, the method further includes: The virtual bus driver sends a first notification message indicating that the virtual driver of the first microphone is successfully created to the multimedia framework, wherein the first notification message includes first identification information of the virtual driver of the first microphone; The first terminal device calls the virtual driver of the first microphone through the first application, including: The multimedia framework detects that the first application is in a startup state, and sends the first identification information to the first application; The first application program calls a virtual driver of the first microphone based on the first identification information.

4. The method according to claim 3, characterized in that The first notification message also includes the creation time of the virtual driver of the first microphone. Before sending the first identification information to the first application, the method also includes: The multimedia framework adds the first identification information to a hardware device list and records the creation time; The multimedia framework determines, based on the creation time, whether the virtual driver of the first microphone is the most recently created driver among the drivers of the device type of microphone in the hardware device list; The sending the first identification information to the first application includes: If so, the multimedia framework sends the first identification information to the first application.

5. The method according to claim 3, characterized in that: The sending the first identification information to the first application includes: The multimedia framework sends a hardware device list to the first application, wherein the hardware device list includes the first identification information; The first application program calls a virtual driver of the first microphone based on the first identification information, including: The first application calls the virtual driver of the first microphone in response to the user's selection operation on the first identification information in the hardware device list, and the selection operation is performed by the user through the interface of the first terminal device displayed on the second terminal device, the interface of the first terminal device includes the interface of the first application, and the interface of the first application includes the hardware device list.

6. The method according to claim 5, characterized in that Before the first terminal device calls the virtual driver of the first microphone through the first application, the method further includes: The first application sends a parameter negotiation request to the virtual bus driver, where the parameter negotiation request is used to request the virtual bus driver to match the audio parameters corresponding to the first application for the virtual driver of the first microphone, and the parameter negotiation request carries parameter requirement information of the first application; The virtual bus driver determines, based on the parameter requirement information, a target parameter corresponding to the virtual driver of the first microphone during the audio data transmission, and sends a request success response corresponding to the parameter negotiation request to the first application; After the first terminal device receives the audio data collected by the second terminal device through the first microphone, the method further includes: The virtual bus driver processes the audio data based on the target parameter through the virtual driver of the first microphone to obtain processed audio data; The virtual bus driver sends the processed audio data to the first application.

7. The method according to any one of claims 1 to 6, characterized in that The system further includes an application server, and the first terminal device and the second terminal device establish a connection through the application server; The first terminal device receives audio data collected by the second terminal device through the first microphone, including: The first terminal device receives, through the first shared service application, audio data collected by the second terminal device through the first microphone and forwarded by the application server.

8. The method according to any one of claims 1 to 6, characterized in that The wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are both in the turned-on state, and the first terminal device and the second terminal device establish a point-to-point P2P connection or a Bluetooth connection; The first terminal device receives audio data collected by the second terminal device through the first microphone, including: The first terminal device receives, through the first communication service manager, audio data collected from the second terminal device through the first microphone.

9. The method according to any one of claims 1 to 8, characterized in that include: The first terminal device receives a first virtual driver deletion request from the second terminal device, where the first virtual driver deletion request is used to request deletion of a virtual driver of the first microphone, and the first virtual driver deletion request includes identification information of the first microphone; The first terminal device deletes the virtual driver of the first microphone through the virtual bus driver based on the first virtual driver deletion request; The first terminal device sends a request success response corresponding to the first virtual drive deletion request to the second terminal device.

10. The method according to claim 9, characterized in that The first terminal device deletes the virtual driver of the first microphone through the virtual bus driver based on the first virtual driver deletion request, including: The first shared service application obtains the first virtual drive deletion request; The first shared service application sends a second virtual driver deletion request to the virtual bus driver through the first device virtualization module, where the second virtual driver deletion request carries first identification information of the virtual driver of the first microphone; The virtual bus driver deletes the virtual driver of the first microphone based on the first identification information.

11. A cross-device hardware calling method, characterized in that: Applied to a system including a first terminal device and a second terminal device, a user logs in to the same account on the first terminal device and the second terminal device, and the first terminal device and the second terminal device have established a connection, the method comprising: The second terminal device sends a virtual driver establishment request to the first terminal device in response to a first operation of the user, wherein the virtual driver establishment request is used to request to establish a virtual driver of a first microphone on the first terminal device, the first microphone is a microphone on the second terminal device, and the first operation is used to collect audio data for the first terminal device through the first microphone; The second terminal device receives a request for using the first microphone from the first terminal device; Based on the request to use the first microphone, the second terminal device collects audio data through the first microphone and sends the audio data to the first terminal device.

12. The method according to claim 11, characterized in that The first operation is a user's selection operation of a preset button in a control interface on the second terminal device.

13. The method according to claim 11 or 12, characterized in that: The method further comprises: The second terminal device stops collecting data through the first microphone in response to a second operation of the user, and sends a first virtual driver deletion request to the first terminal device, wherein the first virtual driver deletion request includes identification information of the first microphone, and the second operation is used to stop collecting audio data for the first terminal device through the first microphone.

14. The method according to claim 13, characterized in that The method further comprises: The second terminal device receives a request response corresponding to the first virtual drive deletion request from the first terminal device.

15. The method according to claim 13 or 14, characterized in that The second operation is a deselection operation of the user on a preset button in the control interface on the second terminal device.

16. The method according to any one of claims 11 to 15, characterized in that The system further includes an application server, and the first terminal device and the second terminal device establish a connection through the application server; The second terminal device sending the audio data to the first terminal device includes: The second terminal device sends the audio data to the application server through a second shared service application program, and the audio data carries the Internet Protocol address of the first terminal device.

17. The method according to any one of claims 11 to 15, characterized in that The wireless fidelity WiFi switch and the Bluetooth switch of the first terminal device and the second terminal device are both in the turned-on state, and the first terminal device and the second terminal device establish a point-to-point P2P connection or a Bluetooth connection; The second terminal device sending the audio data to the first terminal device includes: The second terminal device sends the audio data to the first terminal device through a second communication service manager.

18. A cross-device hardware calling device, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the cross-device hardware calling device executes the method as described in any one of claims 1 to 10, or executes the method as described in any one of claims 11 to 17.

19. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 10, or instructions for implementing the method according to any one of claims 11 to 17.

20. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 10, or is used to implement the method according to any one of claims 11 to 17.

Images