Collaborative call method, device, equipment, storage medium and program product

Abstract

The present application discloses a collaborative call method, apparatus, device, storage medium and program product, which belongs to the field of terminal technology. It includes: if the first device receives a call operation when it is in a multi-screen collaborative state with the second device, a call audio link is established, and the input device of the call audio link is set to the audio acquisition hardware device of the first device, and the output device of the call audio link is set to the audio playback hardware device of the first device. When the first device receives the call operation, it also detects the state of the collaborative call function. If the collaborative call function is in the on state, the input device of the call audio link is switched from the audio acquisition hardware device to the virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to the virtual output module, so as to shut down the audio acquisition hardware device and the audio playback hardware device of the first device during the collaborative call, thereby reducing the hardware power consumption while meeting the collaborative call requirements.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a collaborative call method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of terminal technology, multi-screen collaboration technology has been widely used. Multi-screen collaboration means that after a first device (such as a mobile phone) is connected to a second device (such as a tablet), the screen of the first device is mirrored in the interface of the second device. In this case, the user can make the first device perform the corresponding function by operating the screen of the first device displayed in the interface of the second device.

[0003] In the scenario of multi-screen collaboration, if the first device is in a call, you can switch to the second device to collect and play the call voice, that is, you can make a collaborative call. Specifically, when the first device and the second device are in a collaborative call, the microphone of the second device collects the call voice of the local user and sends it to the first device, and the first device sends the call voice to the remote call device; the remote call device sends the call voice of the remote user to the first device, and the first device sends the call voice to the second device, which is played by the speaker of the second device.

[0004] However, in the related art, during a collaborative call between a first device and a second device, the first device will still turn on its own hardware such as a microphone, a speaker, and an earpiece, which will result in power consumption. Summary of the invention

[0005] The present application provides a collaborative call method, device, equipment, storage medium and program product, which can turn off hardware such as microphones, speakers and receivers during collaborative calls to reduce hardware power consumption. The technical solution is as follows:

[0006] In the first aspect, a collaborative call method is provided. In the method, a first device receives a call operation, wherein the first device and the second device are in a multi-screen collaborative state. Afterwards, the first device establishes a call audio link in response to the call operation, and sets the input device of the call audio link to the audio acquisition hardware device of the first device, and sets the output device of the call audio link to the audio playback hardware device of the first device, and the first device detects the state of the collaborative call function. If it is detected that the collaborative call function is in the on state, the input device of the call audio link is switched from the audio acquisition hardware device to the virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to the virtual output module.

[0007] When the first device and the second device are in a multi-screen collaboration state, the screen of the first device is displayed on the interface of the second device. In this case, the user can make the first device perform a corresponding function by operating the screen of the first device displayed in the interface of the second device.

[0008] The first device can start a collaborative call to collect and play the call audio through the second device. For example, when the call operation is triggered by the user through the screen of the first device displayed on the second device, the first device turns on the collaborative call function when receiving the call operation. Alternatively, before receiving the call operation, the first device can automatically turn on the collaborative call function when detecting that some specific modes are turned on during multi-screen collaboration with the second device.

[0009] The audio acquisition hardware device of the first device is hardware for collecting audio, such as a microphone, etc. The audio playback hardware device of the first device is hardware for playing audio, such as a speaker or a receiver, etc.

[0010] After the collaborative call function is turned on on the first device, since the collection and playback of call voice are performed on the second device, the first device no longer has the need to collect and play call voice. Therefore, in the present application, when the first device detects that the collaborative call function is turned on, it can switch the input device of the call voice from the audio collection hardware device to the virtual input module, and switch the output device of the call voice from the audio playback hardware device to the virtual output module, so as to turn off the audio collection hardware device and the audio playback hardware device during the collaborative call process to reduce power consumption.

[0011] The virtual input module and the virtual output module are both virtual software modules, not hardware devices. The virtual input module does not perform audio capture, such as the virtual input module can be an IN_PROXY module, and the virtual output module does not perform audio playback, such as the virtual output module can be an OUT_PROXY module. In this way, after the first device switches the input device of the call voice to the virtual input module, the first device no longer captures audio, and after the first device switches the output device of the call voice to the virtual output module, the first device no longer plays audio, thereby meeting the needs of collaborative calls while also reducing hardware power consumption.

[0012] It is worth noting that when the first device receives a call operation, it detects the state of the collaborative call function. If it detects that the collaborative call function is in the off state, the input device and output device of the call audio link will not be switched, and the input device of the call audio link will still be the audio acquisition hardware device of the first device, and the output device of the call audio link will be the audio playback hardware device of the first device. If the collaborative call function is in the off state, it means that the first device needs to collect and play the call voice, so it is necessary to keep the input device of the call audio link as the audio acquisition hardware device of the first device, and keep the output device of the call audio link as the audio playback hardware device of the first device, so as to ensure the normal progress of the call.

[0013] For example, when the call operation is triggered by the user on the first device, the first device turns off the collaborative call function upon receiving the call operation. Alternatively, before receiving the call operation, the first device can automatically turn off the collaborative call function when detecting that some specific modes are turned off during the multi-screen collaboration with the second device.

[0014] In one possible scenario, after the first device establishes a call audio link, if it detects that the collaborative call function is switched from an on state to an off state, the input device of the call audio link is switched from a virtual input module to an audio acquisition hardware device, and the output device of the call audio link is switched from a virtual output module to an audio playback hardware device.

[0015] After the first device and the second device have a collaborative call, the collaborative call can also be turned off so that the first device can continue to collect and play the call audio. For example, after the first device establishes a call audio link, if it detects that the user turns off the collaborative call switch displayed by the first device or the second device, the collaborative call function is switched from the on state to the off state.

[0016] After the first device turns off the collaborative call, the collection and playback of the call voice will be switched back to the first device for execution. In this case, the first device has the need to collect and play the call voice. Therefore, in this application, when the first device detects that the collaborative call function is switched from the on state to the off state, it can switch the input device of the call audio link from the virtual input module to the audio collection hardware device, and switch the output device of the call audio link from the virtual output module to the audio playback hardware device, so as to turn on the audio collection hardware device for audio collection, and turn on the audio playback hardware device for audio playback, thereby ensuring the normal progress of the call.

[0017] In one possible scenario, after the first device establishes a call audio link, if it detects that the collaborative call function is switched from an off state to an on state, the input device of the call audio link is switched from an audio acquisition hardware device to a virtual input module, and the output device of the call audio link is switched from an audio playback hardware device to a virtual output module.

[0018] After turning off the collaborative call, the first device can turn on the collaborative call again to switch to the second device for voice collection and playback. For example, after the first device establishes a call audio link, if it detects that the user turns on the collaborative call switch displayed on the first device or the second device, the collaborative call function is switched from the off state to the on state.

[0019] After the first device and the second device conduct a collaborative call, since the collection and playback of call voice are performed on the second device, the first device no longer has the need to collect and play the call voice. Therefore, in the present application, when the first device detects that the collaborative call function is switched from an off state to an on state, the input device of the call audio link can be switched from the audio collection hardware device to a virtual input module, and the output device of the call audio link can be switched from the audio playback hardware device to a virtual output module, so as to shut down the audio collection hardware device and the audio playback hardware device during the collaborative call and reduce power consumption.

[0020] It is worth noting that in the present application, the first device can also turn off the collaborative call function when the call ends. For example, the first device receives a call end operation, and in response to the call end operation, the first device turns off the collaborative call function, and sets the input device of the call audio link to the audio acquisition hardware device, and sets the output device of the call audio link to the audio playback hardware device.

[0021] The call ending operation is an operation for indicating the end of a call, for example, the call ending operation may be an operation for hanging up the phone. The call ending operation may be triggered directly by the user in the first device, for example, the user may click the hang-up button in the call interface in the first device to trigger the call ending operation. Alternatively, the call ending operation may be triggered by the user through the screen of the first device displayed by the second device, for example, the user may click the hang-up button in the call interface of the first device displayed by the second device to trigger the call ending operation.

[0022] From the above description, it can be seen that if the first device conducts a collaborative call during a call, the first device will shut down the audio playback hardware device and the audio playback hardware device. In this case, after establishing the call audio link, when the collaborative call function is turned on, the first device receives the call uplink voice data collected by the second device and sent by the second device, sends the call uplink voice data to the remote call device, and receives the call downlink voice data sent by the remote call device, and sends the call downlink voice data to the second device for playback, so as to realize the collaborative call with the second device.

[0023] If the first device does not conduct a coordinated call during a call, for example, the first device has not started a coordinated call after starting the call, or the first device starts the coordinated call function and then turns off the coordinated call during the call, the first device will turn on the audio playback hardware device and the audio playback hardware device. In this case, after establishing the call audio link, the first device, when the coordinated call function is turned off, collects the call uplink voice data through the audio collection hardware device, sends the call uplink voice data to the remote call device, and receives the call downlink voice data sent by the remote call device, and plays the call downlink voice data through the audio playback hardware device, so as to realize the normal conduct of the call.

[0024] In a second aspect, a collaborative call device is provided, which has the function of implementing the collaborative call method described in the first aspect. The collaborative call device includes at least one module, which is used to implement the collaborative call method described in the first aspect.

[0025] In a third aspect, a collaborative call device is provided, wherein the structure of the collaborative call device includes a processor and a memory, wherein the memory is used to store a program that supports the collaborative call device to execute the collaborative call method provided in the first aspect, and to store data involved in implementing the collaborative call method described in the first aspect. The processor is configured to execute the program stored in the memory. The collaborative call device may also include a communication bus, which is used to establish a connection between the processor and the memory.

[0026] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer executes the collaborative call method described in the first aspect.

[0027] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the collaborative call method described in the first aspect.

[0028] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0030] Figure 2 is a block diagram of a software system of a terminal provided in an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of a call data flow in a non-multi-screen collaboration scenario provided by an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of a call data flow in a multi-screen collaboration scenario provided by an embodiment of the present application;

[0033] Figure 5 This is a schematic diagram of an interface of a tablet computer provided in an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of an interface of a mobile phone provided in an embodiment of the present application;

[0035] Figure 7 is a schematic diagram of an interface of another tablet computer provided in an embodiment of the present application;

[0036] Figure 8 is a schematic diagram of another mobile phone interface provided in an embodiment of the present application;

[0037] Fig. 9 This is a schematic diagram of an interface in a multi-screen collaboration scenario provided by an embodiment of the present application;

[0038] Fig.10 This is another interface diagram in a multi-screen collaboration scenario provided by an embodiment of the present application;

[0039] Fig.11 is a schematic diagram of a multi-screen collaborative system provided in an embodiment of the present application;

[0040] Fig.12 is a flow chart of a collaborative call method provided in an embodiment of the present application;

[0041] Fig.13 is a flowchart of a collaborative call process provided by an embodiment of the present application;

[0042] Fig.14 It is a structural diagram of a collaborative call device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0044] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0045] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" that appear in the differences in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. In addition, the terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0046] The terminal involved in the embodiment of the present application is described below.

[0047] Figure 1 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 1The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identity module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0048] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0049] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0050] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0051] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0052] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal 100. While the charging management module 140 is charging the battery 142, it may also power the terminal 100 through the power management module 141.

[0053] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193 and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0054] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0055] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0056] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0057] The terminal 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0058] The terminal 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0059] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0060] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the terminal 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0061] The terminal 100 can implement audio functions, such as music playing and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0062] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0063] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal 100 can support 1 or N SIM card interfaces, where N is an integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100.

[0064] Next, the software system of the terminal 100 will be described.

[0065] The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the terminal 100.

[0066] Figure 2 1 is a block diagram of a software system of a terminal 100 provided in an embodiment of the present application. Figure 2 The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided from top to bottom into an application layer, an application framework layer, an Android runtime (Android runtime) and a system layer, an extension layer, and a kernel layer.

[0067] The application layer can include a series of application packages. Figure 2 As shown, the application package may include multi-screen collaboration, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, short message and other applications. The multi-screen collaboration application is used to enable the multi-screen collaboration function.

[0068] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2As shown, the application framework layer may include an audio framework (Audio Framework), a distributed mobile sensing development platform (DMSDP), a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0069] The audio framework is responsible for the output of playback data, the collection of recording data, and the comprehensive management of audio transactions. DMSDP is used to provide functional support for the multi-screen collaborative process, such as realizing collaborative calls in the multi-screen collaborative process. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application. The display interface can be composed of one or more views, for example, including a view for displaying a text message notification icon, a view for displaying text, and a view for displaying pictures. The phone manager is used to provide communication functions for the terminal 100, such as the management of call status (including connected, hung up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be a notification that appears in the system's top status bar in the form of an icon or scroll bar text, such as notifications from applications running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message in the status bar, a beep, an electronic device vibrating, an indicator light flashing, etc.

[0070] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0071] The system layer can include multiple functional modules, such as audio services, surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. Audio services are the makers of audio strategies, responsible for the strategy selection of audio device switching, volume adjustment strategy, etc., and are the executors of audio strategies, responsible for the management of input and output stream devices and the processing and transmission of audio stream data. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports a variety of commonly used audio and video format playback and recording, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0072] The extension layer can also be called the hardware abstraction layer (HAL), which can encapsulate the kernel driver, provide an interface to the upper layer, and shield the implementation details of the lower layer. The extension layer connects to the Android Runtime and Framework upwards and to the driver downwards. The extension layer can include the audio hardware abstraction layer (Audio HAL), which is responsible for interacting with audio hardware devices.

[0073] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver, and pulse code modulation (PCM) device.

[0074] The following uses multi-screen collaboration between a mobile phone and a tablet computer as an example to illustrate the application scenarios involved in the embodiments of the present application.

[0075] When making a call or answering an incoming call, the mobile phone can talk to the remote call device. The call can include a voice call, a video call, etc., which is not limited in the embodiments of the present application. For example, the call can be a call established through an operator service, such as a 2G call, a 3G call, a 4G call, a 5G call, etc.

[0076] When a mobile phone makes a call, it involves the collection of uplink voice data and the playback of downlink voice data.

[0077] Uplink voice data refers to the voice data of the local user collected by the mobile phone or the tablet computer that cooperates with the mobile phone during the call. Uplink voice data needs to be sent to the remote call device. Downlink voice data refers to the voice data of the remote user sent by the remote call device and received by the mobile phone during the call. Downlink voice data needs to be played on the mobile phone or the tablet computer that cooperates with the mobile phone.

[0078] When the mobile phone is not performing multi-screen collaboration with the tablet computer, if the mobile phone is making a call, the microphone (Mic) of the mobile phone collects the call voice of the local user and sends the call voice to the remote call device; the remote call device sends the call voice of the remote user to the mobile phone, and the mobile phone plays the call voice through its own speaker (Speaker) or receiver.

[0079] The call data flow in non-multi-screen collaboration scenarios is as follows: Figure 3 As shown, the uplink and downlink voice data of the call flow between the audio chip and devices (microphone, speaker, receiver) of the mobile phone. The audio chip can be ADSP (Audio DSP) and the like. Specifically, the microphone of the mobile phone collects the call voice of the local user and sends the call voice to the audio chip of the mobile phone. The audio chip of the mobile phone sends the call voice to the remote call device through the modem processor of the mobile phone. At the same time, the audio chip of the mobile phone receives the call voice of the remote user sent by the remote call device through the modem processor of the mobile phone, and sends the call voice to the speaker or receiver of the mobile phone for playback.

[0080] When a mobile phone and a tablet computer are in multi-screen collaboration, the mobile phone can switch to the tablet computer to collect and play the call voice, that is, a collaborative call can be made. Specifically, when a mobile phone and a tablet computer are in a collaborative call, the tablet computer's microphone collects the local user's call voice and sends it to the mobile phone, which sends the call voice to the remote call device; the remote call device sends the remote user's call voice to the mobile phone, and the mobile phone sends the call voice to the tablet computer, which is played by the tablet computer's speaker.

[0081] The call data flow in the multi-screen collaboration scenario is as follows: Figure 4As shown, the uplink and downlink voice data of the call flow between the audio chip and Audio HAL of the mobile phone. Specifically, the microphone of the tablet computer collects the call voice of the local user and sends the call voice to the mobile phone. After receiving the call voice, the Audio HAL of the mobile phone sends the call voice to the audio chip of the mobile phone. The audio chip of the mobile phone sends the call voice to the remote call device through the modem processor of the mobile phone. At the same time, the audio chip of the mobile phone receives the call voice of the remote user sent by the remote call device through the modem processor of the mobile phone, and sends the call voice to the Audio HAL of the mobile phone. The Audio HAL of the mobile phone sends the call voice to the tablet computer, which is played by the speaker of the tablet computer.

[0082] In this way, when the mobile phone and the tablet computer are making a collaborative call, the microphone of the mobile phone no longer picks up the sound, and the speaker and receiver of the mobile phone no longer make any sound. In the related art, the mobile phone directly mutes the microphone, speaker and receiver by calling the device mute interface. In this way, the sound collected by the microphone of the mobile phone will not be transmitted, and the speaker and receiver of the mobile phone will not play any sound. However, in this case, the mobile phone will still turn on its own microphone, speaker, receiver and other hardware, which will consume power.

[0083] To this end, an embodiment of the present application provides a collaborative call method, which can directly turn off the microphone, speaker, earpiece and other hardware of the mobile phone when the mobile phone and the tablet computer are conducting a collaborative call, thereby reducing power consumption.

[0084] Several possible connection methods for multi-screen collaboration are described below.

[0085] 1. Establish a connection via Bluetooth.

[0086] For example, if the user wants to work with a mobile phone and a tablet, the Bluetooth in both the mobile phone and the tablet can be turned on first. Then, the user manually turns on the multi-screen collaboration function in the mobile phone. For example, the user can find the "multi-screen collaboration" switch in the mobile phone interface through the path of "Settings"-"More Connections"-"Multi-Screen Collaboration", and set the switch to on to turn on the multi-screen collaboration function of the mobile phone.

[0087] See also Figure 5 The interface diagram of the tablet computer shown in FIG. Figure 5 As shown in Figure (a) in FIG. 1 , the user slides down the status bar of the tablet computer to display the notification panel, which includes the “multi-screen collaboration” option 51. The user clicks the “multi-screen collaboration” option 51, and the tablet computer responds to the user's triggering operation on the “multi-screen collaboration” option 51 by displaying a first prompt window, which includes first operation prompt information for instructing the user on how to operate to achieve multi-screen collaboration. For example, Figure 5 As shown in Figure (b) of FIG. 1 , the first operation prompt information includes the prompt content “1. Turn on the Bluetooth of your mobile phone and bring it close to the machine. After discovering the machine, click “Connect”. 2. After connecting, you can operate the mobile phone on the tablet computer to realize data sharing between devices.” In this way, the user can perform corresponding operations according to the first operation prompt information, such as bringing the mobile phone close to the tablet computer.

[0088] In one example, see Figure 6 As shown in the schematic diagram of the interface of the mobile phone, when the mobile phone approaches the tablet computer and finds the tablet computer, the mobile phone displays a second prompt window, such as Figure 6 As shown in Figure (a) in the figure, the second prompt window includes the prompt content of "Do you want to establish a collaborative connection with the discovered device", as well as the "Connect" option 61 and the "Cancel" option 62. When the user clicks the "Connect" option 61, it means that the user confirms that the collaborative connection is to be established. In response to the user's triggering operation on the "Connect" option 61, the mobile phone establishes a collaborative connection with the tablet computer via Bluetooth. When the user clicks the "Cancel" option 62, it means that the user does not want to establish a collaborative connection. In response to the user's triggering operation on the "Cancel" option 62, the mobile phone does not perform the operation of establishing a collaborative connection. In another example, when the mobile phone is approaching the tablet computer, when the mobile phone discovers the tablet computer, the second prompt window may not be displayed, but a collaborative connection with the tablet computer may be automatically established via Bluetooth.

[0089] As an example but not a limitation, during the process of establishing a collaborative connection between the mobile phone and the tablet computer via Bluetooth, in order to display the progress of establishing the collaborative connection, the mobile phone may also display a third prompt window indicating that the connection is in progress, for example, Figure 6 Optionally, the third prompt window includes a "cancel" option so that the user can cancel the connection at any time when necessary.

[0090] 2. Establish a connection by scanning the QR code.

[0091] For example, the user can find the "Scan code to connect" button in the tablet computer interface through the path of "My phone" - "Connect now" - "Scan code to connect". The user clicks the button, and the tablet computer responds to the user's triggering operation on the button and displays a QR code for establishing a collaborative connection. For example, the following may be displayed: Figure 7 Optionally, the tablet computer may also display a second operation prompting information for prompting the user how to operate to achieve multi-screen collaboration, for example, Figure 7 As shown, the second operation prompt information may be "Use a mobile browser to scan the code to connect."

[0092] In one example, see Figure 8As shown in the schematic diagram of the mobile phone interface, the user can enter the interface with the "Scan" option in the mobile phone browser (or smart vision), for example, Figure 8 The browser interface shown in FIG. (a) in FIG. 1 shows a “Scan” option 81. The user can click the “Scan” option 81, and the mobile phone responds to the user’s triggering operation on the “Scan” option 81, starts the camera, and displays the following Figure 8 The scanning interface shown in Figure (b) in the figure, the user can then aim the camera at the QR code displayed on the tablet computer to scan the code.

[0093] In one example, after the mobile phone scans the code successfully, it sends a request to the tablet computer to establish a collaborative connection. After the tablet computer receives the request sent by the mobile phone, it can display a fourth prompt window, which includes prompt information for prompting the user whether to agree to establish a collaborative connection. For example, the prompt information may include the prompt content "xx device requests to establish a collaborative connection with this end, do you agree to establish a collaborative connection?", as well as an "Agree" option and a "Reject" option. When the user clicks the "Agree" option, it means that the user allows the mobile phone to establish a collaborative connection with the tablet computer, and the tablet computer responds to the user's triggering operation of the "Agree" option and establishes a collaborative connection with the mobile phone. When the user clicks the "Reject" option, it means that the user does not allow the mobile phone to establish a collaborative connection with the tablet computer, and the tablet computer responds to the user's triggering operation of the "Reject" option and notifies the mobile phone that the collaborative connection establishment failed.

[0094] It should be noted that the above is only an example of a user opening a QR code through the path of "My Phone" - "Connect Now" - "Scan Code to Connect" on a tablet computer. Optionally, the QR code can also be opened through other paths. For example, Figure 5 As shown in Figure (b) of FIG. 1 , in addition to the first operation prompt information, the first prompt window also includes the prompt content "Can't find the machine? You can also scan the code to connect", where the four words "Scan the code to connect" are triggerable. The user can click the "Scan the code to connect" content in the first prompt window, and the tablet computer responds to the user's triggering operation on the "Scan the code to connect" content and displays the following Figure 7 In this way, the user can scan the QR code displayed on the tablet computer through the mobile phone, thereby establishing a collaborative connection by scanning the code.

[0095] 3. Establish connection by touching.

[0096] The user can turn on NFC and multi-screen collaboration functions on both the mobile phone and the tablet. After that, the user touches the NFC area on the back of the mobile phone (usually located around the camera on the back of the mobile phone) to the NFC area of ​​the tablet computer (usually located in the lower right corner of the tablet computer), and the mobile phone and the tablet computer respond to the user's touch operation and establish a collaborative connection through NFC. Optionally, before establishing a collaborative connection through NFC, the tablet computer and the mobile phone can also prompt the user whether to agree to establish a collaborative connection. After the user agrees to establish a collaborative connection, the mobile phone and the tablet computer perform the operation of establishing a collaborative connection. In one example, when the mobile phone and the tablet computer successfully establish a collaborative connection, the mobile phone can also remind the user by vibrating or ringing.

[0097] It should be noted that the above several possible connection methods are all described by taking wireless connection as an example. In another embodiment, it can also be implemented by wired connection, such as by Type-C to high definition multimedia interface (HDMI) connection cable, which is not limited in the embodiment of the present application.

[0098] After the mobile phone and tablet computer have successfully established a collaborative connection, Fig. 9 As shown in the figure, the tablet computer will mirror the screen of the mobile phone. In this way, the user can operate the screen of the mobile phone displayed on the tablet computer on the tablet computer according to the needs to make the mobile phone perform corresponding functions. In one example, the mobile phone and the tablet computer synchronously display the following Fig. 9 If the user wants to make a call, he can click the call icon in the main interface of the mobile phone displayed on the tablet computer to open the dial interface of the mobile phone. At this time, the mobile phone and the tablet computer will display the dial interface of the mobile phone simultaneously. Then the user can perform the dialing operation in the dial interface of the mobile phone displayed on the tablet computer to make a call on the mobile phone.

[0099] After making a call on your mobile phone, if the phone starts talking, Fig.10 As shown, the mobile phone and tablet computer will synchronously display the call interface of the mobile phone. When the mobile phone is making a call, you can choose to switch to the tablet computer to collect and play the call voice, that is, you can make a collaborative call. Specifically, when the mobile phone and the tablet computer are making a collaborative call, the microphone of the tablet computer collects the call voice of the local user and sends it to the mobile phone, and the mobile phone sends the call voice to the remote call device; the remote call device sends the call voice of the remote user to the mobile phone, and the mobile phone sends the call voice to the tablet computer, which is played by the speaker of the tablet computer.

[0100] For example, after a mobile phone and a tablet computer have achieved multi-screen collaboration, if the mobile phone starts a call, Fig.10 As shown, after the user pulls down the notification bar of the tablet computer in the tablet computer, the notification bar of the tablet computer may display the prompt content of "has been coordinated to the mobile phone", and the notification bar may also include a switch for switching the call voice to the tablet computer. The user can operate the switch as needed to indicate whether to switch the call voice to the tablet computer, that is, whether to make a collaborative call. Alternatively, the user can also pull down the notification bar of the mobile phone in the mobile phone, and the notification bar of the mobile phone may display the prompt content of "has been coordinated to the tablet computer", and the notification bar may also include a switch for switching the call voice to the tablet computer, and the user can operate the switch as needed to indicate whether to make a collaborative call. If the user operates the switch for switching the call voice to the tablet computer on the tablet computer or the mobile phone during a call, to indicate that the call voice is switched to the tablet computer, that is, to indicate a collaborative call, the collection and playback of the call voice of the ongoing call of the mobile phone is performed on the tablet computer side. If the user operates the switch for switching the call voice to the tablet computer on the tablet computer or the mobile phone during a call, to indicate not to switch the call voice to the tablet computer, that is, to indicate not to make a collaborative call, the collection and playback of the call voice of the ongoing call of the mobile phone is still performed on the mobile phone side.

[0101] It is worth noting that in some embodiments, in the scenario of multi-screen collaboration, the mobile phone can also automatically conduct collaborative calls by default. That is, after the mobile phone and the tablet computer conduct multi-screen collaboration, if the mobile phone starts a call, the collaborative call can be automatically conducted without user operation, that is, the collection and playback of the call voice is automatically switched to the tablet computer side for execution. In this case, if the user does not want to conduct a collaborative call, the user can operate the switch for switching the call voice to the tablet computer on the tablet computer or mobile phone to indicate not to switch the call voice to the tablet computer, that is, to indicate to turn off the collaborative call and switch the call voice back to the mobile phone. At this time, the collection and playback of the call voice of the ongoing call on the mobile phone will be switched back to the mobile phone side for continued execution.

[0102] When a mobile phone and a tablet computer are in a collaborative call, the call voice is switched to the tablet computer side, and it is expected that the mobile phone side will no longer make any sound. To this end, the embodiment of the present application provides a collaborative call method, which can turn off the microphone, speaker, receiver and other hardware of the mobile phone during the collaborative call, thereby reducing power consumption.

[0103] The collaborative call method provided in the embodiment of the present application can be applied to a multi-screen collaborative system, and the multi-screen collaborative system is described below.

[0104] Fig.11 is a schematic diagram of a multi-screen collaboration system provided by an embodiment of the present application. Fig.11 , the multi-screen collaborative system may include a first device 1101 and a second device 1102.

[0105] The first device 1101 and the second device 1102 may communicate via a wired connection or a wireless connection.

[0106] The first device 1101 and the second device 1102 may both be terminals, and the terminals may be Figure 1-Figure 2 The terminal described in the embodiment. For example, the terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a television, etc., and the embodiments of the present application are not limited to this.

[0107] The first device 1101 and the second device 1102 can perform multi-screen collaboration. After the first device 1101 and the second device 1102 perform multi-screen collaboration, the screen image of the first device 1101 can be displayed on the interface of the second device 1102.

[0108] The first device 1101 and the second device 1102 may be terminals of different types or terminals of the same type, which is not limited in the present embodiment. For example, both may be terminals such as mobile phones or tablet computers.

[0109] In a possible implementation, the screen size of the first device 1101 is smaller than the screen size of the second device 1102, so that when the small screen and the large screen perform multi-screen collaboration, the screen image of the small screen is displayed as a window to the interface of the large screen, so that the user can operate the screen image of the small screen in the interface of the large screen, thereby improving the user's operating experience. For example, the first device 1101 is a mobile phone, and the second device 1102 is a tablet computer or a TV. Alternatively, the first device 1101 is a tablet computer, and the second device 1102 is a TV. Of course, the screen size of the first device 1101 can also be larger than the screen size of the second device 1102. For example, the first device 1101 is a tablet computer, and the second device 1102 is a mobile phone.

[0110] When the first device 1101 and the second device 1102 perform multi-screen collaboration, if the first device 1101 starts a call, the collaborative call can be started to switch to the second device 1102 to collect and play the call audio.

[0111] The collaborative call method provided in the embodiment of the present application is applied to the scenario where the first device 1101 and the second device 1102 perform multi-screen collaboration. In this case, the first device 1101 can perform multi-screen collaboration with the second device 1102 by executing the collaborative call method provided in the embodiment of the present application. When the collaborative call is started, the microphone, speaker, and receiver of the first device 1101 can be turned off to reduce power consumption.

[0112] The collaborative call method provided in the embodiment of the present application is described below.

[0113] The collaborative call method provided in the embodiment of the present application is applied to the first device, and the first device and the second device can perform multi-screen collaboration. Multi-screen collaboration refers to displaying the screen image of the first device on the interface of the second device.

[0114] After the first device and the second device perform multi-screen collaboration, the screen of the first device is displayed on the interface of the second device. In this case, the user can make the first device perform the corresponding function by operating the screen of the first device displayed in the interface of the second device. The first device and the second device can achieve multi-screen collaboration in a variety of possible ways. For example, multi-screen collaboration can be achieved through Bluetooth, code scanning, touch, etc. These methods have been described in detail above, and the embodiments of this application will not be repeated here.

[0115] The first device is a device that can make calls. One or more SIM cards may be installed in the first device, and the first device may use any one of the one or more SIM cards to make calls. For example, in the case where the first device has only one SIM card installed, the first device may directly use this SIM card to make calls, such as using this SIM card to make calls, or using this SIM card to answer calls. In the case where multiple SIM cards are installed in the first device, the first device may use one of the multiple SIM cards to make calls, such as using this SIM card to make calls, or using this SIM card to answer calls.

[0116] Next, the call scenarios and call voice switching scenarios involved in the case of multi-screen collaboration between the first device and the second device are explained.

[0117] Call scenario:

[0118] When the first device and the second device perform multi-screen collaboration, the user can directly operate in the first device to start a call with the first device, or the user can operate in the second device to start a call with the first device.

[0119] By way of example, taking dialing a phone number in the first device to start a call as an example, several possible operation modes are described.

[0120] In a first possible operation mode, if the user wants to make a call in the first device, the user can directly perform an operation in the first device to achieve it.

[0121] For example, the user can click on the dialing icon in the main interface of the first device to open the dialing interface of the first device, and then the user can perform a dialing operation in the dialing interface of the first device to dial a phone in the first device to start a call. In the case where only one SIM card is installed in the first device, the user can directly perform a dialing operation in the dialing interface of the first device, so that the SIM card is used to make a call in the first device; in the case where multiple SIM cards are installed in the first device, the user can first select a SIM card in the dialing interface of the first device, and then perform a dialing operation in the dialing interface of the first device, so that the selected SIM card is used to make a call in the first device.

[0122] In the second possible operation mode, if the user wants to make a call on the first device, the operation can be performed on the second device that performs multi-screen collaboration with the first device to achieve it.

[0123] For example, after the first device and the second device perform multi-screen collaboration, the first device and the second device synchronously display the main interface of the first device, and the user can click the icon of making a call in the main interface of the first device displayed on the second device to open the dialing interface of the first device. At this time, the first device and the second device will synchronously display the dialing interface of the first device, and then the user can perform a dialing operation in the dialing interface of the first device displayed on the second device to make a call in the first device to start a call. In particular, when the first device is only equipped with one SIM card, the user can directly perform a dialing operation in the dialing interface of the first device displayed on the second device, so that the SIM card can be used to make a call in the first device; when the first device is equipped with multiple SIM cards, the user can first select a SIM card in the dialing interface of the first device displayed on the second device, and then perform a dialing operation in the dialing interface of the first device displayed on the second device, so that the selected SIM card can be used to make a call in the first device.

[0124] In a third possible operation mode, if the user wants to make a call in the first device, the operation can be performed on both the first device and the second device that performs multi-screen collaboration with the first device to achieve it.

[0125] For example, after the first device and the second device perform multi-screen collaboration, the first device and the second device synchronously display the main interface of the first device, and the user can click the icon of making a call in the main interface of the first device displayed on the second device to open the dialing interface of the first device. At this time, the first device and the second device will synchronously display the dialing interface of the first device, and then the user can perform a dialing operation in the dialing interface of the first device to make a call in the first device to start a call. Among them, in the case where only one SIM card is installed in the first device, the user can directly perform a dialing operation in the dialing interface of the first device, so that the SIM card is used to make a call in the first device; in the case where multiple SIM cards are installed in the first device, the user can first select a SIM card in the dialing interface of the first device, and then perform a dialing operation in the dialing interface of the first device, so that the selected SIM card is used to make a call in the first device.

[0126] For another example, after the first device and the second device perform multi-screen collaboration, the first device and the second device synchronously display the main interface of the first device, and the user can click the dialing icon in the main interface of the first device to open the dialing interface of the first device. At this time, the first device and the second device will synchronously display the dialing interface of the first device, and then the user can perform a dialing operation in the dialing interface of the first device displayed on the second device to make a call in the first device to start a call. In particular, when the first device is only equipped with one SIM card, the user can directly perform a dialing operation in the dialing interface of the first device displayed on the second device, so that the SIM card can be used to make a call in the first device; when the first device is equipped with multiple SIM cards, the user can first select a SIM card in the dialing interface of the first device displayed on the second device, and then perform a dialing operation in the dialing interface of the first device displayed on the second device, so that the selected SIM card can be used to make a call in the first device.

[0127] By way of example, taking answering an incoming call in the first device to start a call as an example, several possible operation modes are described.

[0128] In a first possible operation mode, when there is an incoming call on the first device, if the user wants to answer the incoming call on the first device, the user can directly perform an operation on the first device to achieve the answer.

[0129] For example, if there is an incoming call on the first device, the first device will display an incoming call interface, and the user can click the answer button in the incoming call interface of the first device to answer the incoming call in the first device to start the call. Wherein, if only one SIM card is installed in the first device, the incoming call in the first device is a call for this SIM card, so the SIM card is used in the first device to answer the call; if multiple SIM cards are installed in the first device, the incoming call in the first device is a call for one of the multiple SIM cards, so the SIM card is used in the first device to answer the call.

[0130] The second possible operation mode is that when there is an incoming call on the first device, if the user wants to answer the call on the first device, the operation can be performed on the second device that performs multi-screen collaboration with the first device to achieve it.

[0131] For example, after the first device and the second device perform multi-screen collaboration, if there is an incoming call on the first device, the first device and the second device will synchronously display the incoming call interface of the first device, and the user can click the answer button in the incoming call interface of the first device displayed on the second device to answer the incoming call in the first device to start the call. Wherein, in the case where only one SIM card is installed in the first device, the incoming call in the first device is a call for this SIM card, so this SIM card is used to answer the call in the first device; in the case where multiple SIM cards are installed in the first device, the incoming call in the first device is a call for one SIM card among the multiple SIM cards, so this SIM card is used to answer the call in the first device.

[0132] Call voice switching scenario:

[0133] In the case where the first device and the second device perform multi-screen collaboration, the first device and the second device both include a collaborative call switch, which is used to indicate whether the second device collects and plays the call voice when the first device is on a call, that is, whether to turn on the collaborative call function. For example, the pull-down notification bar of the first device and the pull-down notification bar of the second device can both include a collaborative call switch, so that the user can turn on or off the collaborative call switch in the first device or the second device. Of course, the collaborative call switch can also be set in other interfaces of the first device and the second device, and the embodiments of the present application are not limited to this.

[0134] When the collaborative call switch is turned on, the collaborative call function is turned on. In this case, if the first device is making a call with the second device during multi-screen collaboration, a collaborative call will be made during the call on the first device, that is, the call voice will be collected and played through the second device during the call on the first device. Specifically, when the first device and the second device are making a collaborative call, the microphone of the second device collects the call voice of the local user and sends it to the first device, and the first device sends the call voice to the remote call device; the remote call device sends the call voice of the remote user to the first device, and the first device sends the call voice to the second device, which is played by the speaker of the second device.

[0135] When the collaborative call switch is turned off, the collaborative call function is turned off. In this case, if the first device is making a call with the second device during multi-screen collaboration, the collaborative call will not be made during the call on the first device, that is, the first device will still collect and play the call voice during the call on the first device. Specifically, the microphone of the first device collects the call voice of the local user and sends it to the remote call device; the remote call device sends the call voice of the remote user to the first device, which is played by the speaker of the first device.

[0136] In some embodiments, when the first device and the second device just start multi-screen collaboration, the collaborative call function is turned on by default, that is, when the first device and the second device establish a multi-screen collaborative connection, the collaborative call function is turned on. In other embodiments, during the multi-screen collaboration between the first device and the second device, the activation of the collaborative call function can also be triggered by a specific mode association. For example, if the user plays audio or video in the second device, the first device can activate the collaborative call function. In this case, when the collaborative call function is activated, the first device and the second device can also turn on the collaborative call switch synchronously, and when the collaborative call is turned off, the first device and the second device can also turn off the collaborative call switch synchronously.

[0137] As an example, if the user wants to turn off the collaborative call function, the user can directly turn off the collaborative call switch in the first device or directly turn off the collaborative call switch in the second device during the multi-screen collaboration between the first device and the second device to turn off the collaborative call function. If the user wants to turn on the collaborative call function after turning it off, the user can directly turn on the collaborative call switch in the first device or directly turn on the collaborative call switch in the second device during the multi-screen collaboration between the first device and the second device to turn on the collaborative call function.

[0138] For example, the collaborative call function has been turned on during the multi-screen collaboration between the first device and the second device. If the first device has not started a call yet and the user wants to turn off the collaborative call function, the collaborative call switch can be turned off directly in the first device or the second device. In this way, if the first device starts a call later, the first device will still collect and play the call audio.

[0139] Alternatively, the collaborative call function has been turned off during the multi-screen collaboration between the first device and the second device. If the first device has not started a call yet and the user wants to turn on the collaborative call function, the collaborative call switch can be turned on directly in the first device or the second device. In this way, if the first device subsequently starts a call, the second device will collect and play the call audio.

[0140] For another example, the collaborative call function has been turned on during the multi-screen collaboration between the first device and the second device. If the first device has started a call, the second device is now collecting and playing the call audio. If the user wants to turn off the collaborative call function, he can directly turn off the collaborative call switch in the first device or the second device. In this way, the call audio of the call being conducted by the first device will be switched to the first device for collection and playback.

[0141] Alternatively, the collaborative call function has been turned off during the multi-screen collaboration between the first device and the second device. If the first device has started a call, the first device is responsible for collecting and playing the call audio. If the user wants to turn on the collaborative call function, he or she can directly turn on the collaborative call switch in the first device or the second device. In this way, the call audio of the ongoing call on the first device will be switched to the second device for collection and playback.

[0142] As another example, the collaborative call function can be turned on and off by the user's behavior of making a call or answering a call. Specifically, if during the multi-screen collaboration between the first device and the second device, the user makes a call or answers a call directly on the first device, the collaborative call function will be turned off, that is, the first device will collect and play the call voice during the call on the first device, and the collaborative call switches in the first device and the second device will also be in the off state. Alternatively, if during the multi-screen collaboration between the first device and the second device, the user makes a call or answers a call on the first device by operating the screen of the first device displayed on the second device, the collaborative call function will be turned on, that is, the second device will collect and play the call voice during the call on the first device, and the collaborative call switches in the first device and the second device will also be in the on state.

[0143] From the above description, it can be seen that the user can turn on and off the collaborative call function in two ways.

[0144] Specifically, if the first device receives a first instruction triggered by the user during multi-screen collaboration with the second device, the collaborative call function can be turned off. The first instruction is used to instruct the first device to collect and play the call voice when the first device is on a call. Optionally, the first instruction is triggered when the user turns off the collaborative call switch in the first device or the second device; or, the first instruction is triggered when the user makes a call or answers a call on the first device.

[0145] If the first device receives a second instruction triggered by the user during multi-screen collaboration with the second device, the collaborative call function can be enabled. The second instruction is used to instruct the second device to collect and play the call voice when the first device is making an operational call. Optionally, the second instruction is triggered when the user turns on the collaborative call switch in the first device or the second device; or, the second instruction is triggered when the user operates the screen of the first device displayed on the second device to make the first device make a call or answer an incoming call.

[0146] In addition, the first device may also automatically enable the collaborative call function. For example, when the first device establishes a multi-screen collaborative connection with the second device, the collaborative call function may be automatically enabled; or, when the first device detects that some specific modes are enabled during the multi-screen collaboration with the second device, the collaborative call function may be automatically enabled.

[0147] The first device may also automatically turn off the collaborative call function. For example, when the first device detects that some specific modes are turned off during multi-screen collaboration with the second device, the collaborative call function may be automatically turned off; or when the first device ends the call, that is, when the call is hung up, the collaborative call function may be automatically turned off.

[0148] Next, the overall process of the collaborative call method provided in the embodiment of the present application is described.

[0149] Fig.12 is a flow chart of a collaborative call method provided by an embodiment of the present application. Fig.12 , the method comprising:

[0150] Step 1201: The first device and the second device perform multi-screen collaboration.

[0151] After the first device and the second device perform multi-screen collaboration, the first device and the second device are in a multi-screen collaboration state, and the screen of the first device is displayed on the interface of the second device. In this case, the user can make the first device perform the corresponding function by operating the screen of the first device displayed in the interface of the second device. The first device and the second device can achieve multi-screen collaboration in a variety of possible ways. For example, multi-screen collaboration can be achieved through Bluetooth, code scanning, touch, etc. These methods have been described in detail above, and the embodiments of this application will not be repeated here.

[0152] Step 1202: The first device receives a call operation.

[0153] The call operation is an operation for instructing the first device to make a call, such as the call operation can be an operation of making a call or answering an incoming call. The call operation can be triggered by the user directly in the first device, or the call operation can be triggered by the user through the screen of the first device displayed by the second device. The various possible triggering methods of the call operation have been described in detail above, and the embodiments of the present application will not be repeated here.

[0154] Step 1203: In response to the call operation, the first device establishes a call audio link, sets the input device of the call audio link to the audio acquisition hardware device of the first device, and sets the output device of the call audio link to the audio playback hardware device of the first device.

[0155] After receiving the call operation, the first device determines that it needs to make a call, and therefore needs to establish a call audio link, and sets the input device and output device of the call audio link. The input device of the call audio link is a device for collecting uplink voice data of the call, and the output device of the call audio link is a device for playing downlink voice data of the call. When the first device receives the call operation, it sets the input device of the call audio link to its own audio acquisition hardware device to collect audio through the audio acquisition hardware device, and sets the output device of the call audio link to its own audio playback hardware device to play audio through the audio playback hardware device.

[0156] The audio acquisition hardware device of the first device is hardware for collecting audio, such as a microphone, etc. The audio playback hardware device of the first device is hardware for playing audio, such as a speaker or a receiver, etc.

[0157] Step 1204: The first device detects the status of the collaborative call function. If it is detected that the collaborative call function is in the turned-on state, the input device of the call audio link is switched from the audio acquisition hardware device to the virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to the virtual output module.

[0158] When receiving a call operation, the first device may also detect the state of the collaborative call function to determine whether to switch the input device and output device of the call audio link according to the state of the collaborative call function.

[0159] The first device can enable the collaborative call function to collect and play the call voice through the second device. The first device can enable the collaborative call function in a variety of possible ways, which have been described in detail above and will not be repeated in the embodiments of the present application.

[0160] For example, when the call operation is triggered by the screen of the first device displayed by the user on the second device, the first device turns on the collaborative call function when receiving the call operation. Alternatively, before receiving the call operation, the first device can automatically turn on the collaborative call function when detecting that some specific modes are turned on during the multi-screen collaboration with the second device.

[0161] After the first device turns on the collaborative call function, the collection and playback of call voice will be switched to the second device for execution. Specifically, during the collaborative call between the first device and the second device, the second device collects the uplink voice data of the call of the local user (such as the second device can collect the uplink voice data of the call through its own microphone), and sends the uplink voice data of the call to the first device, and the first device sends the uplink voice data of the call to the remote call device. At the same time, the remote call device collects the downlink voice data of the call of the remote user, and sends the downlink voice data of the call to the first device, and the first device sends the downlink voice data of the call to the second device, which is played by the second device (such as the second device can play the downlink voice data of the call through its own speaker).

[0162] After the first device and the second device conduct a collaborative call, since the collection and playback of call voice are performed on the second device, the first device no longer has the need to collect and play the call voice. Therefore, in an embodiment of the present application, when the first device detects that the collaborative call function is in the on state, the input device of the call audio link can be switched from the audio collection hardware device to the virtual input module, and the output device of the call audio link can be switched from the audio playback hardware device to the virtual output module, so as to turn off the audio collection hardware device and the audio playback hardware device during the collaborative call to reduce power consumption.

[0163] Both the virtual input module and the virtual output module are virtual software modules, not hardware devices. The virtual input module does not perform audio capture, such as the virtual input module can be an IN_PROXY module, and the virtual output module does not perform audio playback, such as the virtual output module can be an OUT_PROXY module. In this way, after the first device switches the input device of the call audio link to the virtual input module, the first device no longer captures audio, and after the first device switches the output device of the call audio link to the virtual output module, the first device no longer plays audio, thereby meeting the needs of collaborative calls while reducing hardware power consumption.

[0164] It is worth noting that when the first device receives a call operation, it detects the state of the collaborative call function. If it detects that the collaborative call function is in the off state, the input device and output device of the call audio link will not be switched, and the input device of the call audio link will still be the audio acquisition hardware device of the first device, and the output device of the call audio link will be the audio playback hardware device of the first device. If the collaborative call function is in the off state, it means that the first device needs to collect and play the call voice, so it is necessary to keep the input device of the call audio link as the audio acquisition hardware device of the first device, and keep the output device of the call audio link as the audio playback hardware device of the first device, so as to ensure the normal progress of the call.

[0165] For example, when the call operation is triggered by the user on the first device, the first device turns off the collaborative call function upon receiving the call operation. Alternatively, before receiving the call operation, the first device can automatically turn off the collaborative call function when detecting that some specific modes are turned off during the multi-screen collaboration with the second device.

[0166] In some embodiments, if the first device turns off the collaborative call function during a call, the following step 1205 can be continued.

[0167] Step 1205: After the first device establishes the call audio link, if it detects that the collaborative call function is switched from the on state to the off state, the input device of the call audio link is switched from the virtual input module to the audio acquisition hardware device, and the output device of the call audio link is switched from the virtual output module to the audio playback hardware device.

[0168] After the first device and the second device have a collaborative call, the collaborative call can also be turned off so that the first device can continue to collect and play the call audio. The first device can turn off the collaborative call in a variety of possible ways, which have been described in detail above and will not be repeated in the embodiments of the present application. For example, after the first device establishes a call audio link, if it detects that the user turns off the collaborative call switch displayed by the first device or the second device, the collaborative call function is switched from the on state to the off state.

[0169] After the first device turns off the collaborative call, the collection and playback of the call voice will be switched back to the first device for execution. In this case, the first device has the need to collect and play the call voice. Therefore, in the embodiment of the present application, when the first device detects that the collaborative call function is switched from the on state to the off state, it can switch the input device of the call audio link from the virtual input module to the audio collection hardware device, and switch the output device of the call audio link from the virtual output module to the audio playback hardware device, so as to turn on the audio collection hardware device for audio collection, and turn on the audio playback hardware device for audio playback, thereby ensuring the normal conduct of the call.

[0170] In some embodiments, if the first device turns on the collaborative call function during a call, the following step 1206 can be continued.

[0171] Step 1206: After the first device establishes the call audio link, if it detects that the collaborative call function is switched from the off state to the on state, the input device of the call audio link is switched from the audio acquisition hardware device to the virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to the virtual output module.

[0172] After turning off the collaborative call, the first device can turn on the collaborative call again to switch to the second device for collecting and playing the call voice. The first device can turn on the collaborative call in a variety of possible ways, which have been described in detail above and will not be repeated in the embodiments of the present application. For example, after the first device establishes a call audio link, if it detects that the user turns on the collaborative call switch displayed by the first device or the second device, the collaborative call function is switched from the off state to the on state.

[0173] After the first device and the second device conduct a collaborative call, since the collection and playback of call voice are performed on the second device, the first device no longer has the need to collect and play the call voice. Therefore, in an embodiment of the present application, when the first device detects that the collaborative call function is switched from an off state to an on state, the input device of the call audio link can be switched from the audio collection hardware device to a virtual input module, and the output device of the call audio link can be switched from the audio playback hardware device to a virtual output module, so as to shut down the audio collection hardware device and the audio playback hardware device during the collaborative call and reduce power consumption.

[0174] It is worth noting that in the embodiment of the present application, the first device can also turn off the collaborative call function when the call ends. For example, the first device receives a call end operation, and in response to the call end operation, the first device turns off the collaborative call function, and sets the input device of the call audio link to the audio acquisition hardware device, and sets the output device of the call audio link to the audio playback hardware device.

[0175] The call ending operation is an operation for indicating the end of a call, for example, the call ending operation may be an operation for hanging up the phone. The call ending operation may be triggered directly by the user in the first device, for example, the user may click the hang-up button in the call interface in the first device to trigger the call ending operation. Alternatively, the call ending operation may be triggered by the user through the screen of the first device displayed by the second device, for example, the user may click the hang-up button in the call interface of the first device displayed by the second device to trigger the call ending operation.

[0176] If the first device does not conduct a collaborative call before receiving the call end operation, that is, the collaborative call function is in the off state, then when the first device receives the call end operation, the collaborative call function is kept in the off state, and the input device of the call audio link is kept as the audio acquisition hardware device, and the output device of the call audio link is kept as the audio playback hardware device. If the first device conducts a collaborative call before receiving the call end operation, that is, the collaborative call function is in the on state, then when the first device receives the call end operation, the collaborative call function is switched from the on state to the off state, and the input device of the call audio link is switched from the virtual input device to the audio acquisition hardware device, and the output device of the call audio link is switched from the virtual output device to the audio playback hardware device.

[0177] From the above description, it can be seen that if the first device conducts a collaborative call during a call, the first device will shut down the audio playback hardware device and the audio playback hardware device. In this case, after establishing the call audio link, when the collaborative call function is turned on, the first device receives the call uplink voice data collected by the second device and sent by the second device, sends the call uplink voice data to the remote call device, and receives the call downlink voice data sent by the remote call device, and sends the call downlink voice data to the second device for playback, so as to realize the collaborative call with the second device.

[0178] If the first device does not conduct a collaborative call during a call, for example, the first device has not started a collaborative call after starting the call, or the first device starts the collaborative call function and then turns off the collaborative call during the call, the first device will turn on the audio playback hardware device and the audio playback hardware device. In this case, after the first device establishes a call audio link, when the collaborative call function is in the off state, the first device collects the uplink voice data of the local user's call through the audio collection hardware device, and sends the uplink voice data to the remote call device. At the same time, the remote call device collects the downlink voice data of the remote user's call, and sends the downlink voice data to the first device. After receiving the downlink voice data sent by the remote call device, the first device plays the downlink voice data through the audio playback hardware device to ensure the normal progress of the call.

[0179] For ease of understanding, the following Figure 2 The software system shown and Fig.13 The collaborative call method is illustrated by using the flowchart of the collaborative call process shown in FIG.

[0180] See also Figure 2 , the software system of the first device may include a call application and a multi-screen collaboration application in the application layer, an audio framework and DMSDP in the application framework layer, an audio service in the system layer, an Audio HAL in the extension layer, and a PCM device in the kernel layer. For example, the Audio HAL may be a PrimaryAudio HAL. In addition, the first device also includes an audio chip, which can communicate with a remote call device. For example, the audio chip can communicate with the remote call device through a modem processor, that is, the uplink voice data of the call is sent to the remote call device through the modem processor, and the downlink voice data of the call is received by the modem processor.

[0181] See also Fig.13 The collaborative call process may include the following steps 1301-1311.

[0182] Step 1301: The call application detects an operation of making a call or answering an incoming call in the call interface, and instructs the audio service to call the Audio HAL to establish a call audio link through the audio framework.

[0183] When the call application detects an operation of making a call or answering an incoming call in the call interface (i.e., the incall UI interface), it is determined that the first device needs to make a call, and thus the call establishment process can be started. At this time, the call application can instruct the audio service to call the Audio HAL through the audio framework, so that the Audio HAL starts the voice call process and establishes the call, as described in step 1302.

[0184] Step 1302: The Audio HAL instructs the audio chip to establish a call audio link, and sets the input device of the call audio link to the microphone of the first device, and sets the output device of the call audio link to the earpiece of the first device.

[0185] After the Audio HAL starts the voice call process, it selects a sound device and performs route switching to establish a call audio link. Selecting a sound device instructs the audio chip to set the input device of the call audio link to the microphone of the first device, and to set the output device of the call audio link to the handset of the first device. Performing route switching instructs the audio chip to send the uplink voice data of the call collected by the microphone of the first device to the remote call device, and to send the downlink voice data of the call received from the remote call device to the handset of the first device for playback.

[0186] The operation of setting the input device of the call audio link to the microphone of the first device and the output device of the call audio link to the earpiece of the first device in step 1302 is the same as the operation of setting the input device of the call audio link to the audio acquisition hardware device of the first device and the output device of the call audio link to the audio playback hardware device of the first device in the above step 1203, and will not be repeated in this embodiment of the present application.

[0187] Step 1303: When the first device and the second device are in the multi-screen collaborative state, the multi-screen collaborative application monitors the call state, and if the call state is monitored to change from idle to in-call, determines whether the collaborative call function is turned on.

[0188] The multi-screen collaboration application will continuously monitor the call status of the first device during the multi-screen collaboration process. If it monitors that the call status changes from idle to offhook, it is determined that the first device has started a call. At this time, the multi-screen collaboration application can determine whether the collaborative call function is turned on. If the collaborative call function is turned on, a collaborative call needs to be made during this call; if the collaborative call switch function is turned off, a collaborative call does not need to be made during this call.

[0189] When the collaborative call function is turned on, it means that the user allows the establishment of a collaborative call. When the collaborative call function is turned off, it means that the user does not allow the establishment of a collaborative call.

[0190] In one possible manner, the user can manually turn on or off the collaborative call function. For example, the user can click the collaborative call switch to turn on and off the collaborative call switch to turn on and off the collaborative call function.

[0191] In another possible way, the opening and closing of the collaborative call function can also be determined by the user's behavior of making a call or answering a call. Specifically, if the user makes a call or answers a call directly on the first device during the multi-screen collaboration between the first device and the second device, the first device will automatically turn off the collaborative call function. Alternatively, if the user makes a call or answers a call on the first device by operating the screen of the first device displayed on the second device during the multi-screen collaboration between the first device and the second device, the first device will automatically turn on the collaborative call function.

[0192] Step 1304: When the collaborative call function is enabled, the multi-screen collaborative application sends a first switching instruction to the DMSDP to instruct the DMSDP to establish a collaborative call.

[0193] The first switching instruction is used to instruct the DMSDP to establish a collaborative call, that is, to instruct to switch the call voice to the second device so that the second device can collect and play the call voice.

[0194] It is worth noting that the above steps 1303 and 1304 are only explained by taking the example of sending the first switching instruction to DMSDP when the multi-screen collaborative application monitors that the call state changes from idle to in-call and the collaborative call function is turned on. Optionally, the multi-screen collaborative application can also send the first switching instruction to DMSDP in other situations. For example, when the multi-screen collaborative application detects that the collaborative call function switches from an off state to an on state, and the call state of the first device monitored is in-call, it can also send the first switching instruction to DMSDP.

[0195] Step 1305: After receiving the first switching instruction sent by the multi-screen collaborative application, if the DMSDP determines that the call status monitored by itself is in progress, it calls the Audio HAL to establish a collaborative call.

[0196] During the collaborative call, DMSDP will also continue to monitor the call status of the first device. After receiving the first switching instruction, DMSDP determines that the multi-screen collaborative application indicates that a collaborative call needs to be established. At this time, DMSDP will determine whether to establish a collaborative call based on the call status it monitors.

[0197] Specifically, when DMSDP determines that the call state monitored by itself is idle, if it determines that the received first switching instruction does not match the call state monitored by itself, then the collaborative call is not established. When DMSDP determines that the call state monitored by itself is in call, if it determines that the received first switching instruction matches the call state monitored by itself, then the Audio HAL is called to establish the collaborative call, as described in step 1306.

[0198] Step 1306: Audio HAL creates a Stream flow corresponding to the uplink voice data of the call to open the corresponding path in the PCM device and the audio chip for obtaining the uplink voice data of the call, and creates a Stream flow corresponding to the downlink voice data of the call to open the corresponding path in the PCM device and the audio chip for obtaining the downlink voice data of the call, so as to establish a collaborative call.

[0199] In this case, during a collaborative call between the first device and the second device, DMSDP receives uplink voice data of the call collected by the second device and sent by the second device, DMSDP sends the uplink voice data of the call to AudioHAL, Audio HAL writes the uplink voice data of the call to the PCM device, the PCM device sends the uplink voice data of the call to the audio chip, and the audio chip sends the uplink voice data of the call to the remote call device through the corresponding path.

[0200] At the same time, the audio chip receives the call downlink voice data sent by the remote call device through the corresponding channel, and sends the call downlink voice data to the PCM device. The Audio HAL reads the call downlink voice data from the PCM device and sends the call downlink voice data to the DMSDP. The DMSDP sends the call downlink voice data to the second device, which plays it.

[0201] Step 1307: After establishing the collaborative call, the Audio HAL instructs the audio chip to switch the input device of the call audio link from the microphone to the IN_PROXY module, and to switch the output device of the call audio link from the earpiece to the OUT_PROXY module.

[0202] After the first device and the second device have a collaborative call, since the collection and playback of the call voice are both performed on the second device, the first device no longer has the need to collect and play the call voice. Therefore, in the embodiment of the present application, after starting the collaborative call, the first device can switch the input device of the call audio link from the microphone of the first device to the IN_PROXY module, and switch the output device of the call audio link from the earpiece of the first device to the OUT_PROXY module, so as to turn off the microphone and earpiece of the first device during the collaborative call. In this way, the first device no longer collects and plays audio, thereby reducing hardware power consumption while meeting the needs of the collaborative call.

[0203] The operation of switching the input device of the call audio link from the microphone to the IN_PROXY module and the output device of the call audio link from the earpiece to the OUT_PROXY module in step 1307 is the same as the operation of switching the input device of the call audio link from the audio acquisition hardware device to the virtual input module and the output device of the call audio link from the audio playback hardware device to the virtual output module in the above step 1204, and will not be repeated in this embodiment of the present application.

[0204] Step 1308: When the first device and the second device are in a multi-screen collaborative state, if the multi-screen collaborative application monitors that the call state changes from being in call to being idle, or detects that the collaborative call function changes from being on to being off, then a second switching instruction is sent to the DMSDP to instruct to close the collaborative call.

[0205] The second switching instruction is used to instruct the DMSDP to close the collaborative call, that is, to instruct to switch the call voice back to the first device so that the first device can collect and play the call voice.

[0206] If the multi-screen collaborative application monitors that the call status of the first device changes from in-call to idle, it means that the first device has ended the call, and at this time, an instruction can be given to close the collaborative call. For example, after the user hangs up the call, the call status of the first device changes from in-call to idle. At this time, the multi-screen collaborative application monitors the change of the call status and can instruct to close the collaborative call.

[0207] If the multi-screen collaborative application detects that the collaborative call function is turned from on to off, it means that the user wants to collect and play the call voice on the first device, and then the collaborative call can be turned off. For example, the user can manually click the collaborative call switch to turn off the collaborative call function. At this time, after the multi-screen collaborative application detects that the collaborative call switch is turned from on to off, it can instruct to turn off the collaborative call.

[0208] Step 1309: After receiving the second switching instruction sent by the multi-screen collaborative application, the DMSDP calls the AudioHAL to close the collaborative call.

[0209] After receiving the second switching instruction, the DMSDP determines that the multi-screen collaborative application indicates that the collaborative call needs to be ended. Therefore, at this time, the DMSDP can call the Audio HAL to close the collaborative call, as described in step 1310.

[0210] Step 1310: The Audio HAL closes the Stream corresponding to the uplink voice data of the call to close the corresponding path in the PCM device and the audio chip used to obtain the uplink voice data of the call, and closes the Stream corresponding to the downlink voice data of the call to close the corresponding path in the PCM device and the audio chip used to obtain the downlink voice data of the call, thereby closing the collaborative call.

[0211] After the collaborative call is closed, the collection and playback of the call voice are restored to the original state, that is, switched back to the first device for execution. In this case, if the first device is still in a call after closing the collaborative call, the first device itself needs to collect and play the call voice.

[0212] Step 1311: While closing the collaborative call, the Audio HAL instructs the audio chip to switch the input device of the call audio link from the IN_PROXY module to its own microphone, and to switch the output device of the call audio link from the OUT_PROXY module to its own earpiece.

[0213] In this case, if the first device is still in a call, the microphone of the first device collects the uplink voice data of the call and sends it to the audio chip, and the audio chip sends the uplink voice data of the call collected by the microphone of the first device to the remote call device. The remote call device sends the downlink voice data of the call to the audio chip, and the audio chip sends the received downlink voice data of the call sent by the remote call device to the handset of the first device for playback.

[0214] In step 1311, the operation of switching the input device of the call audio link from the IN_PROXY module to its own microphone, and switching the output device of the call audio link from the OUT_PROXY module to its own listening device is the same as the operation of switching the input device of the call audio link from the virtual input module to the audio acquisition hardware device, and switching the output device of the call audio link from the virtual output module to the audio playback hardware device in the above step 1205, and the embodiments of the present application will not be repeated in detail.

[0215] In an embodiment of the present application, when the first device and the second device are in a multi-screen collaborative state, if the first device receives a call operation, a call audio link is established, and the input device of the call audio link is set to the audio acquisition hardware device of the first device, and the output device of the call audio link is set to the audio playback hardware device of the first device. At the same time, when the first device receives the call operation, it can also detect the state of the collaborative call function. If the collaborative call function is in the on state, the input device of the call audio link is switched from the audio acquisition hardware device to a virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to a virtual output module, so as to shut down the audio acquisition hardware device and the audio playback hardware device during the collaborative call. In this way, the first device no longer collects audio or plays audio, thereby reducing hardware power consumption while meeting the collaborative call requirements.

[0216] Fig.14 is a schematic diagram of a structure of a collaborative call device provided in an embodiment of the present application. The device can be implemented by software, hardware or a combination of both to become part or all of a computer device. The computer device can be Figure 1-Figure 2 The terminal shown in the embodiment. Fig.14 The device includes: a receiving module 1401, an establishing module 1402, and a switching module 1403.

[0217] The receiving module 1401 is used to receive a call operation, wherein the apparatus and the second device are in a multi-screen collaborative state;

[0218] An establishing module 1402 is used to establish a call audio link in response to a call operation, and set the input device of the call audio link to the audio acquisition hardware device of the device, and set the output device of the call audio link to the audio playback hardware device of the device;

[0219] The switching module 1403 is used to switch the input device of the call audio link from the audio acquisition hardware device to the virtual input module, and switch the output device of the call audio link from the audio playback hardware device to the virtual output module if it is detected that the collaborative call function is in the turned-on state. The collaborative call function is used to instruct the second device to collect and play the call voice, the virtual input module does not perform audio collection, and the virtual output module does not perform audio playback.

[0220] Optionally, the switching module 1403 is further configured to:

[0221] After the call audio link is established, if it is detected that the collaborative call function is switched from the on state to the off state, the input device of the call audio link is switched from the virtual input module to the audio acquisition hardware device, and the output device of the call audio link is switched from the virtual output module to the audio playback hardware device.

[0222] Optionally, the switching module 1403 is further configured to:

[0223] After the call audio link is established, if it is detected that the collaborative call function is switched from the off state to the on state, the input device of the call audio link is switched from the audio acquisition hardware device to the virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to the virtual output module.

[0224] Optionally, the device further comprises:

[0225] The first call module is used to receive call uplink voice data collected by the second device and sent by the second device after the call audio link is established and the collaborative call function is turned on, and send the call uplink voice data to the remote call device, as well as receive call downlink voice data sent by the remote call device, and send the call downlink voice data to the second device for playback.

[0226] Optionally, the device further comprises:

[0227] The second call module is used to collect call uplink voice data through the audio collection hardware device after the call audio link is established and the collaborative call function is in the off state, and send the call uplink voice data to the remote call device, as well as receive the call downlink voice data sent by the remote call device and play the call downlink voice data through the audio playback hardware device.

[0228] Optionally, the device further comprises:

[0229] an activation module, configured to activate the collaborative call function in response to the call operation if the call operation is triggered by the user through a screen of the device displayed by the second device;

[0230] The closing module is used to close the collaborative call function in response to the call operation if the call operation is triggered by the user on the device.

[0231] Optionally, the device further comprises:

[0232] A closing module, configured to switch the collaborative call function from an on state to an off state if it is detected that the user turns off the collaborative call switch displayed by the apparatus or the second device after the call audio link is established;

[0233] The enabling module is used to switch the collaborative call function from an off state to an on state after establishing a call audio link if it is detected that the user turns on the collaborative call switch displayed by the device or the second device.

[0234] Optionally, the device further comprises:

[0235] The receiving module 1401 is further used to receive a call ending operation;

[0236] The setting module is used to close the collaborative call function in response to the call ending operation, and set the input device of the call audio link to the audio acquisition hardware device, and set the output device of the call audio link to the audio playback hardware device.

[0237] Optionally, the audio acquisition hardware device is a microphone, and the audio playback hardware device is a speaker or a receiver.

[0238] In an embodiment of the present application, when in a multi-screen collaborative state with a second device, if a call operation is received, a call audio link is established, and the input device of the call audio link is set to its own audio acquisition hardware device, and the output device of the call audio link is set to its own audio playback hardware device. At the same time, when the call operation is received, the state of the collaborative call function can also be detected. If the collaborative call function is in the on state, the input device of the call audio link is switched from the audio acquisition hardware device to a virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to a virtual output module, so as to shut down the audio acquisition hardware device and the audio playback hardware device during the collaborative call process. In this way, it no longer collects audio or plays audio, thereby reducing hardware power consumption while meeting the collaborative call requirements.

[0239] It should be noted that: the collaborative call device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example for collaborative calls. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0240] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0241] The collaborative call device and the collaborative call method embodiments provided in the above embodiments belong to the same concept. The specific working process of the units and modules in the above embodiments and the technical effects brought about can be found in the method embodiment part and will not be repeated here.

[0242] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0243] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A collaborative conversation method, characterized in that: The method is applied to a first device, the first device includes a call application, a multi-screen collaboration application, a distributed mobile perception platform DMSDP, an audio hardware abstraction layer Audio HAL and an audio chip, and the method includes: The call application receives a call operation, wherein the first device and the second device are in a multi-screen collaborative state, and a collaborative call function is turned on when the first device and the second device establish a multi-screen collaborative connection, and the collaborative call function is used to instruct the second device to collect and play the call voice; In response to the call operation, the call application calls the Audio HAL to establish a call audio link; the Audio HAL instructs the audio chip to establish a call audio link, and sets the input device of the call audio link to the audio acquisition hardware device of the first device, and sets the output device of the call audio link to the audio playback hardware device of the first device, so as to open the audio acquisition hardware device and the audio playback hardware device, the input device of the call audio link is a device for collecting call uplink voice data, and the output device of the call audio link is a device for playing call downlink voice data; the audio chip receives the call uplink voice data collected by the input device of the call audio link and sent by the input device of the call audio link, and sends the call uplink voice data to the remote call device, and the audio chip receives the call downlink voice data sent by the remote call device, and sends the call downlink voice data to the output device of the call audio link for playback; The multi-screen collaborative application monitors the call status when the first device and the second device are in the multi-screen collaborative state, and if the call status is monitored to change from idle to in-call, sends a first switching instruction to the DMSDP when the collaborative call function is turned on, so as to instruct the DMSDP to establish a collaborative call; After the DMSDP receives the first switching instruction sent by the multi-screen collaborative application, if it determines that the call status monitored by itself is in call, the Audio HAL is called to establish a collaborative call; the Audio HAL creates a Stream corresponding to the call uplink voice data to open the pulse code modulation PCM device for obtaining the call uplink voice data and the corresponding path in the audio chip, and creates a Stream corresponding to the call downlink voice data to open the PCM device for obtaining the call downlink voice data and the corresponding path in the audio chip to achieve the establishment of a collaborative call, and after the collaborative call is established, instructs the audio chip to switch the input device of the call audio link from the audio acquisition hardware device to the virtual input module, and switches the output device of the call audio link from the audio playback hardware device to the virtual output module, so as to close the audio acquisition hardware device and the audio playback hardware device during the collaborative call, the virtual input module and the virtual output module are virtual software modules, the virtual input module does not perform audio acquisition, the virtual output module does not perform audio playback, the virtual input module is an IN_PROXY module, and the virtual output module is an OUT_PROXY module; During the collaborative call, the DMSDP receives the call uplink voice data collected by the second device and sent by the second device, and sends the call uplink voice data to the Audio HAL; the Audio HAL writes the call uplink voice data to the PCM device; the PCM device sends the call uplink voice data to the audio chip; the audio chip sends the call uplink voice data to the remote call device through the corresponding path; During the collaborative call, the audio chip receives the call downlink voice data sent by the remote call device through the corresponding path, and sends the call downlink voice data to the PCM device; the Audio HAL reads the call downlink voice data from the PCM device, and sends the call downlink voice data to the DMSDP; the DMSDP sends the call downlink voice data to the second device, and the second device plays it; The method further comprises: When the first device and the second device are in the multi-screen collaborative state, if the multi-screen collaborative application detects that the collaborative call function is switched from the on state to the off state, the multi-screen collaborative application sends a second switching instruction to the DMSDP to instruct the DMSDP to turn off the collaborative call; After receiving the second switching instruction sent by the multi-screen collaborative application, the DMSDP calls the AudioHAL to close the collaborative call; the Audio HAL closes the Stream corresponding to the call uplink voice data to close the PCM device used to obtain the call uplink voice data and the corresponding path in the audio chip, and closes the Stream corresponding to the call downlink voice data to close the PCM device used to obtain the call downlink voice data and the corresponding path in the audio chip to achieve the closure of the collaborative call, and instructs the audio chip to switch the input device of the call audio link from the virtual input module to the audio acquisition hardware device, and switches the output device of the call audio link from the virtual output module to the audio playback hardware device, so as to open the audio acquisition hardware device and the audio playback hardware device.

2. The method according to claim 1, characterized in that The method further comprises: After establishing the call audio link, if the first device detects that the collaborative call function is switched from an off state to an on state, the input device of the call audio link is switched from the audio acquisition hardware device to the virtual input module, and the output device of the call audio link is switched from the audio playback hardware device to the virtual output module.

3. The method according to claim 1 or 2, characterized in that The method further comprises: If the call operation is triggered by the user through the screen of the first device displayed by the second device, then in response to the call operation, the first device activates the collaborative call function; If the call operation is triggered by the user on the first device, then in response to the call operation, the first device turns off the collaborative call function.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: After the call audio link is established, if the first device detects that the user turns off the collaborative call switch displayed by the first device or the second device, the collaborative call function is switched from an on state to an off state; After establishing the call audio link, if the first device detects that the user turns on the collaborative call switch displayed by the first device or the second device, the collaborative call function is switched from an off state to an on state.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: The first device receives a call ending operation; In response to the call ending operation, the first device turns off the collaborative call function, sets the input device of the call audio link to the audio acquisition hardware device, and sets the output device of the call audio link to the audio playback hardware device.

6. The method according to claim 1 or 2, characterized in that: The audio acquisition hardware device is a microphone, and the audio playback hardware device is a speaker or a receiver.

7. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by the processor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 6.

Images