A multi-screen cooperation call method, system, terminal and storage medium

Abstract

Embodiments of the present application relate to the technical field of terminals, and provide a multi-screen cooperative call method and system, a terminal and a storage medium. When a first terminal and a second terminal are in multi-screen cooperation, if a user operates a first terminal interface displayed by the second terminal to answer an incoming call, the call sound can be switched from the first terminal to the second terminal, that is, the sound of the other party in the call is played through the second terminal, and meanwhile, the second terminal collects the sound of the user and sends the sound to the other terminal through the first terminal, so that the user communicates with the other party in the call through the second terminal. At this time, if the user starts a recording function, the recording file includes the sound of the other party in the call sent by the other terminal and the sound of the user collected by the second terminal.

Description

[0001] The application is a divisional application, the parent application number is 202111519288X, the parent application name is a multi-screen cooperative call method and system, terminal and storage medium, the parent application date is December 14, 2021. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the terminal field, in particular, a multi-screen cooperative call method, system, terminal and storage medium. BACKGROUND

[0003] With the development of electronic technology and mobile Internet, a user or a family may have multiple terminals at the same time, such as mobile phones, notebook computers, tablet computers, smart home devices, etc. In the prior art, multiple terminals can cooperate with each other to realize multi-screen cooperation across devices, thereby providing continuous service experience for users.

[0004] In the prior art, multi-screen cooperation between multiple terminals can only be used for making calls by terminals with SIM cards, for example, when a mobile phone and a tablet computer are in multi-screen cooperation, if the SIM card is installed in the mobile phone, the user can only make calls by the mobile phone and cannot hear the voice of the other party in the call by the tablet computer. SUMMARY

[0005] Embodiments of the present application aim to provide a multi-screen cooperative call method, system, terminal and storage medium, to solve the problem of how to switch the call sound from the mobile phone to the tablet / computer when the operator call is made in the multi-screen cooperation state.

[0006] In a first aspect, the embodiments of the present application provide a multi-screen cooperative call method, the method comprising: receiving, by a first terminal, an incoming call of a call opposite device when a second terminal is connected to the first terminal and displays an interface of the first terminal; in response to a user's call answering operation on the interface of the first terminal in the second terminal, receiving, by the first terminal, call downlink data sent by the call opposite device and sending the call downlink data to the second terminal, and playing, by the second terminal, the call downlink data; collecting, by the second terminal, first call uplink data and sending the first call uplink data to the call opposite device through the first terminal; and in response to a user's opening recording operation, obtaining, by the first terminal, recording data, wherein the recording data comprises the call downlink data and the first call uplink data.

[0007] The method for multi-screen cooperation call provided in the embodiments of the present application can switch the call sound from the first terminal to the second terminal if the user operates the first terminal interface displayed by the second terminal to answer the incoming call during the multi-screen cooperation between the first terminal and the second terminal, that is, the sound of the other party of the call is played through the second terminal, meanwhile, the user's sound collected by the second terminal is sent to the other party of the call through the first terminal, so that the user can communicate with the other party of the call through the second terminal. At this time, if the user opens the recording function, the recording file includes the sound of the other party of the call sent by the other party of the call and the sound of the user collected by the second terminal.

[0008] Optionally, the method further includes: in the case that the first terminal is connected with the second terminal and the second terminal displays the first terminal interface, in response to the user's dialing operation on the first terminal interface in the second terminal, the first terminal calls the other party of the call; after the other party of the call answers, the first terminal receives the call downlink data sent by the other party of the call and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data; the second terminal collects the first call uplink data and sends the first call uplink data to the other party of the call through the first terminal.

[0009] In the embodiments of the present application, if the user operates the first terminal interface displayed by the second terminal to dial a call during the multi-screen cooperation between the first terminal and the second terminal, the call sound can be switched from the first terminal to the second terminal, so that the user can communicate with the other party of the call through the second terminal.

[0010] Optionally, the method further includes: in the case that the first terminal is in call with the other party of the call, the first terminal displays a pop-up box, wherein the pop-up box is used to prompt the user to perform multi-screen cooperation between the first terminal and the second terminal; in response to the user's operation on the connection icon in the pop-up box, the first terminal is connected with the second terminal and the second terminal displays the first terminal interface, the first terminal receives the call downlink data sent by the other party of the call and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data; the second terminal collects the first call uplink data and sends the first call uplink data to the other party of the call through the first terminal.

[0011] In the embodiments of the present application, if the user connects the first terminal and the second terminal and opens the multi-screen cooperation when the user communicates with the other party of the call through the first terminal, the call sound can be switched from the first terminal to the second terminal, so that the user can communicate with the other party of the call through the second terminal.

[0012] Optionally, the method further comprises: in the case that the first terminal is in a call with the call opposite device, and the first terminal is connected with the second terminal and the second terminal displays the first terminal interface, the first terminal displays a first notification panel; in response to a user operation on an audio / video switching icon in the first notification panel in the first terminal, the first terminal receives call downlink data sent by the call opposite device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data; and the second terminal collects first call uplink data and sends the first call uplink data to the call opposite device through the first terminal.

[0013] In the embodiments of the present application, when a user is in a call with the other party through the first terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the first terminal to switch the audio / video, the call sound can be switched from the first terminal to the second terminal, so that the user is in a call with the other party through the second terminal.

[0014] Optionally, the method further comprises: in the case that the first terminal is in a call with the call opposite device, and the first terminal is connected with the second terminal and the second terminal displays the first terminal interface, the second terminal displays a first notification panel; in response to a user operation on an audio / video switching icon in the first notification panel in the second terminal, the first terminal receives call downlink data sent by the call opposite device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data; and the second terminal collects first call uplink data and sends the first call uplink data to the call opposite device through the first terminal.

[0015] In the embodiments of the present application, when a user is in a call with the other party through the first terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the second terminal to switch the audio / video, the call sound can be switched from the first terminal to the second terminal, so that the user is in a call with the other party through the second terminal.

[0016] Optionally, the first terminal comprises a call application program, a cooperation module, a virtual call hardware management module, a transmission module, a first PCM device, a second PCM device, an audio processor and an audio device; and the second terminal comprises a transmission module and an audio device.

[0017] The first terminal receives the call downlink data sent by the call opposite terminal device and sends to the second terminal, and the second terminal plays the call downlink data, including: the call application program sends an opening virtual call instruction to the cooperative module in response to the user's operation; the cooperative module sends the opening virtual call instruction to the virtual call hardware management module; in response to receiving the opening virtual call instruction, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and prohibits the first PCM device from acquiring uplink data; the audio processor receives the call downlink data sent by the call opposite terminal device; the first PCM device acquires the call downlink data from the audio processor; the virtual call hardware management module reads the call downlink data from the first PCM device and sends the call downlink data to the transmission module of the first terminal; the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, and the audio device of the second terminal plays the call downlink data.

[0018] The second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal device through the first terminal, including: the audio device of the second terminal collects the first call uplink data, and the transmission module of the second terminal sends the first call uplink data to the transmission module of the first terminal; the transmission module of the first terminal sends the first call uplink data to the virtual call hardware management module; the virtual call hardware management module writes the first call uplink data into the second PCM device; the audio processor reads the first call uplink data from the second PCM device and sends to the call opposite terminal device.

[0019] In the embodiments of the present application, by adding a cooperative module, a virtual call hardware management module and a transmission module in the first terminal, and adding a transmission module in the second terminal, the call sound is switched from the first terminal to the second terminal, so that the user can call with the other party through the second terminal.

[0020] Optionally, the first terminal further comprises: an audio service class, an audio hardware management module and a virtual call buffer area; in response to the user's opening recording operation, the first terminal acquires recording data, including: the call application program sends an opening recording instruction to the audio service class in response to the opening recording operation; the audio service class sends the opening recording instruction to the audio hardware management module; in response to receiving the opening recording instruction, the audio hardware management module acquires the recording data from the virtual call buffer area, wherein the recording data is stored after the virtual call hardware management module mixes the call downlink data and the first call uplink data.

[0021] In the embodiments of the present application, the voice of the other party of the call and the voice of the user collected by the second terminal are mixed and stored in the virtual call buffer area by the virtual call hardware management module, and the audio hardware management module reads the voices of the two parties of the call from the virtual call buffer area, so that the first terminal can record the voices of the two parties of the call when the user calls the other party of the call through the second terminal.

[0022] Optionally, the method further comprises: in the case that the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, and the user calls the opposite terminal device through the second terminal, in response to the user's closing operation on the interface of the first terminal in the second terminal, the first terminal plays the call downlink data and collects second call uplink data to send to the opposite terminal device; and the first terminal obtains recording data, wherein the recording data comprises the call downlink data and the second call uplink data.

[0023] In the embodiments of the present application, in the case that the user calls the other party of the call through the second terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user disconnects the connection between the first terminal and the second terminal, the call voice can be switched from the second terminal back to the first terminal, so that the user calls the other party of the call through the first terminal.

[0024] Optionally, the method further comprises: in the case that the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, and the user calls the opposite terminal device through the second terminal, the first terminal displays a second notification panel; in response to the user's operation on the disconnection icon in the second notification panel in the first terminal, the first terminal plays the call downlink data and collects second call uplink data to send to the opposite terminal device.

[0025] In the embodiments of the present application, in the case that the user calls the other party of the call through the second terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user disconnects the connection between the first terminal and the second terminal, the call voice can be switched from the second terminal back to the first terminal, so that the user calls the other party of the call through the first terminal.

[0026] Optionally, the method further comprises: in the case that the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, and the user calls the opposite terminal device through the second terminal, the first terminal displays a second notification panel; in response to the user's operation on the audio / video switching icon in the second notification panel in the first terminal, the first terminal plays the call downlink data and collects second call uplink data to send to the opposite terminal device.

[0027] In the embodiments of the present application, when the user is in a call with the other party via the second terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the first terminal to switch the audio and video, the call sound can be switched from the second terminal back to the first terminal, so that the user is in a call with the other party via the first terminal.

[0028] Optionally, the method further comprises: when the first terminal is connected with the second terminal, the second terminal displays the interface of the first terminal, and the user is in a call with the other party via the second terminal, the second terminal displays a second notification panel; in response to the user's operation on the audio and video switching icon in the second notification panel in the second terminal, the first terminal plays the call downlink data and collects second call uplink data to send to the other party device.

[0029] In the embodiments of the present application, when the user is in a call with the other party via the second terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the second terminal to switch the audio and video, the call sound can be switched from the second terminal back to the first terminal, so that the user is in a call with the other party via the first terminal.

[0030] Optionally, the first terminal comprises: a call application program, a cooperation module, a virtual call hardware management module, a transmission module, a first PCM device, an audio processor and an audio device.

[0031] The step that the first terminal plays the call downlink data and collects second call uplink data to send to the other party device comprises: the call application program sends a virtual call closing instruction to the cooperation module in response to the user's operation; the cooperation module sends the virtual call closing instruction to the virtual call hardware management module; in response to receiving the virtual call closing instruction, the virtual call hardware management module unblocks the audio device of the first terminal and cancels the prohibition of the first PCM device to acquire uplink data; the audio processor receives the call downlink data and plays it through the audio device of the first terminal, and acquires the second call uplink data collected by the audio device of the first terminal and sends it to the other party device.

[0032] In the embodiment of the present application, the voice of the call is switched back to the first terminal from the second terminal through the added cooperative module and the virtual call hardware management module in the first terminal, so that the user can communicate with the other party of the call through the first terminal. Optionally, the first terminal further comprises an audio hardware management module; the first terminal obtaining the recording data comprises: the audio hardware management module reading the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device from the audio processor by acquiring the call downlink data and the second call uplink data and mixing the data.

[0033] In the embodiment of the present application, after the voice of the call is switched back to the first terminal from the second terminal, the voice of the other party of the call and the voice of the user collected by the first terminal are acquired from the audio processor by the first PCM device, and the voice of the two parties of the call is read from the first PCM device by the audio hardware management module, so as to realize call recording.

[0034] Optionally, the first terminal comprises a call application program, an audio service class, an audio hardware management module, a virtual call buffer area, a virtual call hardware management module, a first PCM device and an audio processor.

[0035] In response to the user's start recording operation, the first terminal obtaining the recording data comprises: the call application program sending a start recording instruction to the audio service class in response to the start recording operation; the audio service class sending the start recording instruction to the audio hardware management module; in response to receiving the start recording instruction, the audio hardware management module judges whether the first terminal is currently in virtual call; if the first terminal is currently in virtual call, the audio hardware management module acquires the recording data from the virtual call buffer area, wherein the recording data is stored after the virtual call hardware management module mixes the call downlink data and the first call uplink data; if the second terminal is not currently in virtual call, the audio hardware management module opens the first PCM device and reads the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device from the audio processor by acquiring the call downlink data and the second call uplink data and mixing the data.

[0036] In the embodiment of the present application, when the first terminal starts recording, if the user is communicating with the other party of the call through the second terminal, the recording is the voice of the other party of the call and the voice of the user collected by the second terminal; if the user is communicating with the other party of the call through the first terminal, the recording is the voice of the other party of the call and the voice of the user collected by the first terminal; so as to ensure normal recording of the voice of the two parties of the call.

[0037] Optionally, the method further comprises: the call application sending a close-recording instruction to the audio service class in response to a user's close-recording operation; the audio service class sending the close-recording instruction to the audio hardware management module; in response to receiving the close-recording instruction, the audio hardware management module determining whether the first terminal is currently in a virtual call; if the first terminal is currently in a virtual call, the audio hardware management module does not close the first PCM device and stops obtaining the recording data from the virtual call cache area; if the first terminal is not currently in a virtual call, the audio hardware management module closes the first PCM device to read the recording data and stops obtaining the recording data from the first PCM device.

[0038] In the embodiments of the present application, when the first terminal closes recording, if the user is in a call with the other party through the second terminal, the first PCM device is not closed; if the user is in a call with the other party through the first terminal, the first PCM device is closed; thereby ensuring normal call between the user and the other party.

[0039] Optionally, the virtual call hardware management module maintains a virtual call flag bit, the virtual call flag bit having a first state and a second state, the first state representing that the first terminal is currently in a virtual call, and the second state representing that the first terminal is not currently in a virtual call; the step of the audio hardware management module determining whether the first terminal is currently in a virtual call comprises: the audio hardware management module detecting the virtual call flag bit; if the virtual call flag bit is in the first state, the first terminal is currently in a virtual call; if the virtual call flag bit is in the second state, the first terminal is not currently in a virtual call.

[0040] In the embodiments of the present application, when the first terminal opens or closes recording, the audio hardware management module determines whether the user is in a call with the other party through the first terminal or the second terminal by detecting the virtual call flag bit, thereby ensuring normal recording of the voices of the two parties in the call.

[0041] Optionally, the audio hardware management module maintains a recording flag bit; the method further comprises: in response to receiving the open-recording instruction, the audio hardware management module setting the recording flag bit to a first identifier, the first identifier representing that the first terminal is currently in call recording; in response to receiving the close-recording instruction, the audio hardware management module setting the recording flag bit to a second identifier, wherein the second identifier represents that the first terminal is not currently in call recording.

[0042] In the embodiments of the present application, when the first terminal starts or stops recording, the audio hardware management module resets the recording flag bit, so that when the call sound is switched from the first terminal to the second terminal or from the second terminal to the first terminal, the virtual call hardware management module can accurately determine whether the user is recording, thereby ensuring normal recording of the sound of the two parties in the call.

[0043] Optionally, the first terminal comprises a call application, a cooperation module, a virtual call hardware management module, a transmission module, a first PCM device, a second PCM device, a virtual call cache area and an audio device.

[0044] The first terminal receives the call downlink data sent by the call opposite terminal device and sends it to the second terminal, the second terminal plays the call downlink data, and the second terminal collects the first call uplink data and sends it to the call opposite terminal device through the first terminal, comprising:

[0045] The call application sends an open virtual call instruction to the cooperation module in response to the user's operation;

[0046] The cooperation module sends the open virtual call instruction to the virtual call hardware management module;

[0047] In response to receiving the open virtual call instruction, the virtual call hardware management module determines whether the first terminal is currently recording the call;

[0048] If the first terminal is currently recording the call, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and prohibits the first PCM device from obtaining uplink data, reads the call downlink data from the first PCM device and sends it to the transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal and writes it into the second PCM device, and mixes the call downlink data and the first call uplink data to obtain the recording data and stores it in the virtual call cache area;

[0049] If the first terminal is not currently recording the call, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and prohibits the first PCM device from obtaining uplink data, reads the call downlink data from the first PCM device and sends it to the transmission module of the first terminal, and receives the first call uplink data sent by the transmission module of the first terminal and writes it into the second PCM device.

[0050] In this embodiment of the application, when switching the call audio from the first terminal to the second terminal, if the user is talking to the other party and recording, the audio of the other party is mixed by the virtual call hardware management module and the user's audio collected by the second terminal and stored in the virtual call buffer. This allows the audio hardware management module to read the audio of both parties from the virtual call buffer, thereby enabling the first terminal to continue recording while the user is talking to the other party through the second terminal.

[0051] Optionally, the step of the first terminal playing the downlink call data and collecting the second uplink call data to send to the peer device includes: the call application responding to the user's operation by sending a command to close the virtual call to the collaboration module; the collaboration module sending the command to close the virtual call to the virtual call hardware management module; in response to receiving the command to close the virtual call, the virtual call hardware management module determining whether the first terminal is currently recording a call; if the first terminal is currently recording a call, the virtual call hardware management module unblocking the audio device of the first terminal and unblocking the first PCM device from acquiring uplink data; if the first terminal is not currently recording a call, the virtual call hardware management module unblocking the audio device of the first terminal and shutting down the first PCM device.

[0052] In this embodiment, when switching the call audio from the second terminal back to the first terminal, if the user is talking to and recording a call with the other party, the virtual call hardware management module cancels the restriction on the first PCM device to obtain uplink data. This allows the first PCM device to obtain the voice of the other party and the user's voice collected by the first terminal from the audio processor. Consequently, the audio hardware management module can read the voices of both parties from the first PCM device, thus enabling the first terminal to continue recording while the user is talking to the other party through the first terminal.

[0053] Optionally, the first terminal further includes an audio hardware management module, which maintains a recording flag bit. The recording flag bit has a first identifier and a second identifier. The first identifier indicates that the first terminal is currently recording a call, and the second identifier indicates that the first terminal is not currently recording a call. The step of the virtual call hardware management module determining whether the first terminal is currently recording a call includes: the virtual call hardware management module detecting the recording flag bit; if the recording flag bit is at the first identifier, then the first terminal is currently recording a call; if the recording flag bit is at the second identifier, then the first terminal is not currently recording a call.

[0054] In the embodiments of the present application, when switching the call sound from the first terminal to the second terminal or switching the call sound from the second terminal back to the first terminal, the virtual call hardware management module determines whether the user is recording by detecting the recording flag bit, so that the recording of the sound of the two parties of the call can be continued after the terminal is switched.

[0055] Optionally, the virtual call hardware management module maintains a virtual call flag bit, and the method further comprises: in response to receiving the virtual call starting instruction, the virtual call hardware management module sets the virtual call flag bit to a first state, the first state indicating that the first terminal is currently in virtual call; and in response to receiving the virtual call closing instruction, the virtual call hardware management module sets the virtual call flag bit to a second state, the second state indicating that the first terminal is currently not in virtual call.

[0056] In the embodiments of the present application, when switching the call sound from the first terminal to the second terminal or switching the call sound from the second terminal back to the first terminal, the virtual call hardware management module resets the virtual call flag bit, so that when the first terminal starts or closes recording, the audio hardware management module can accurately determine whether the user is calling the other party of the call through the first terminal or the second terminal, thereby ensuring normal recording of the sound of the two parties of the call.

[0057] In a second aspect, the embodiments of the present application also provide a call method for multi-screen cooperation, applied to a first terminal, the method comprising: the first terminal receiving a call from a call opposite terminal device, wherein the first terminal is connected with a second terminal and the second terminal displays a first terminal interface; in response to a user's call answering operation on the first terminal interface in the second terminal, receiving call downlink data sent by the call opposite terminal device and sending the call downlink data to the second terminal, so that the second terminal plays the call downlink data; receiving first call uplink data collected by the second terminal and sending the first call uplink data to the call opposite terminal device; and in response to a user's recording starting operation, obtaining recording data, wherein the recording data comprises the call downlink data and the first call uplink data.

[0058] In a third aspect, the embodiments of the present application further provide a method for multi-screen cooperative call, applied to a second terminal, and the method comprises the following steps: receiving and playing call downlink data sent by a first terminal, wherein the call downlink data is sent by a call opposite terminal to the first terminal under the condition that the first terminal is connected with the second terminal and the second terminal displays a first terminal interface, the first terminal receives a call from the call opposite terminal, and the call opposite terminal sends the call downlink data to the first terminal in response to a call receiving operation of a user on the first terminal interface in the second terminal; the second terminal collects first call uplink data, and sends the first call uplink data to the call opposite terminal through the first terminal, so that the first terminal obtains recording data in response to an opening recording operation of the user, wherein the recording data comprises the call downlink data and the first call uplink data.

[0059] In a fourth aspect, the embodiments of the present application further provide a terminal, which comprises one or more processors, a memory, an audio device, and a display screen, wherein the memory is used to store computer program codes, the computer program codes comprise computer instructions, and when the processors execute the computer instructions, the terminal executes the multi-screen cooperative call method executed by the first terminal in the first aspect, or the terminal executes the multi-screen cooperative call method executed by the second terminal in the first aspect.

[0060] In a fifth aspect, the embodiments of the present application further provide a multi-screen cooperative call system, which comprises a first terminal and a second terminal, the first terminal executes the multi-screen cooperative call method executed by the first terminal in the first aspect, and the second terminal executes the multi-screen cooperative call method executed by the second terminal in the first aspect.

[0061] In a sixth aspect, the embodiments of the present application further provide a computer readable storage medium, which comprises a computer program, and when the computer program runs on a terminal, the terminal executes the multi-screen cooperative call method executed by the first terminal in the first aspect, or the terminal executes the multi-screen cooperative call method executed by the second terminal in the first aspect.

[0062] The beneficial effects of the second aspect to the sixth aspect and the implementation manners thereof can refer to the description of the beneficial effects of the method of the first aspect and the implementation manners thereof. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 A schematic structural diagram of a multi-screen cooperative call system provided by the embodiments of the present application;

[0064] Figure 2 A structural schematic diagram of a terminal provided by the embodiments of the present application Figure 1 ;

[0065] Figure 3 A software structure block diagram of a terminal provided for an embodiment of the present application;

[0066] Figure 4 A telephone function schematic diagram of a terminal provided for the prior art;

[0067] Figure 5 A flowchart of a multi-screen cooperative call method provided for an embodiment of the present application Figure 1 ;

[0068] Figure 6 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 1 ;

[0069] Figure 7 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 2 ;

[0070] Figure 8 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 3 ;

[0071] Figure 9 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 4 ;

[0072] Figure 10 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 5 ;

[0073] Figure 11 A data flow of a multi-screen cooperative call method provided for an embodiment of the present application Figure 1 ;

[0074] Figure 12 An interaction schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 1 ;

[0075] Figure 13 A flowchart of a multi-screen cooperative call method provided for an embodiment of the present application Figure 2 ;

[0076] Figure 14 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 6 ;

[0077] Figure 15 An application scenario schematic diagram of a multi-screen cooperative call method provided for an embodiment of the present application Figure 7 ;

[0078] Figure 16 This application provides an illustration of an application scenario for a multi-screen collaborative calling method. Figure 8 ;

[0079] Figure 17 This application provides an illustration of an application scenario for a multi-screen collaborative calling method. Figure 9 ;

[0080] Figure 18 An interactive illustration of a multi-screen collaborative call method provided in this application embodiment. Figure 2 ;

[0081] Figure 19 Data flow of a multi-screen collaborative call method provided in the embodiments of this application Figure 2 ;

[0082] Figure 20 Data flow of a multi-screen collaborative call method provided in the embodiments of this application Figure 3 ;

[0083] Figure 21 Data flow of a multi-screen collaborative call method provided in the embodiments of this application Figure 4 ;

[0084] Figure 22 An interactive illustration of a multi-screen collaborative call method provided in this application embodiment. Figure 3 ;

[0085] Figure 23 An interactive illustration of a multi-screen collaborative call method provided in this application embodiment. Figure 4 ;

[0086] Figure 24 A schematic diagram of the structure of a terminal provided in this application embodiment. Figure 2 . Detailed Implementation

[0087] Multi-screen collaboration is a distributed technology that allows multiple devices to work together to achieve cross-device and cross-system multi-screen collaboration. When a mobile phone and tablet / computer are in multi-screen collaboration, the user can control the mobile phone by operating the tablet / computer, for example, by replying to emails or answering / making phone calls.

[0088] Currently, the mobile phone and the tablet / computer multi-screen cooperation, if the mobile phone and the call opposite device perform the operator call, since the operator call needs to rely on the SIM card on the mobile phone, so in the call process, the voice of the other party of the call can only be sent to the mobile phone of the user through the mobile network, and the voice of the user can only be sent to the other party of the call through the mobile phone. That is, the user can complete the action of answering / making a call by operating the tablet / computer, but cannot hear the voice of the other party of the call through the tablet / computer.

[0089] Therefore, when the operator call is performed in the multi-screen cooperation state, how to switch the call voice from the mobile phone to the tablet / computer is a technical problem to be solved by the embodiments of the present application.

[0090] In order to solve the above technical problem, the embodiments of the present application provide a multi-screen cooperation call method, the mobile phone can send the voice of the other party of the call to the tablet / computer for playing, at the same time, the tablet / computer can collect the voice of the user and send it to the call opposite device through the mobile phone, so as to switch the call voice from the mobile phone to the tablet / computer.

[0091] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0092] The multi-screen cooperation call method provided by the embodiments of the present application can be applied to the multi-screen cooperation call system 100 as shown in the Figure 1 As shown in the Figure 2 The multi-screen cooperation call system 100 can include a first terminal 101 and a second terminal 102.

[0093] The first terminal 101 can be a device with display function and telephone function, for example, a mobile phone, a tablet computer, a wearable electronic device, a notebook computer, a smart television, etc., and the embodiments of the present application do not make any limitation on this.

[0094] The second terminal 102 can be a device with display function and sound receiving and playing function, for example, a mobile phone, a tablet computer, a smart television, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a vehicle-mounted device, a virtual reality device, etc., and the embodiments of the present application do not make any limitation on this.

[0095] The first terminal 101 can perform multi-screen cooperation with the second terminal 102, and the first terminal 101 can perform a call with a call opposite terminal device. When the first terminal 101 performs multi-screen cooperation with the second terminal 102, if the first terminal 101 performs a call with the call opposite terminal device, the first terminal 101 can switch the call sound from the first terminal 101 to the second terminal 102, so that the call sound is emitted from the tablet / computer.

[0096] For example, the first terminal 101 can be a mobile phone, the second terminal 102 can be a tablet computer or a notebook computer, or the first terminal 101 can be a tablet computer, the second terminal 102 can be a notebook computer or a smart television, or the first terminal 101 and the second terminal 102 can both be mobile phones, or the first terminal 101 and the second terminal 102 can both be tablet computers, etc.

[0097] For example, the first terminal 101 can be a mobile phone, the second terminal 102 can be a tablet computer or a notebook computer, or the first terminal 101 can be a tablet computer, the second terminal 102 can be a notebook computer or a smart television, or the first terminal 101 and the second terminal 102 can both be mobile phones, or the first terminal 101 and the second terminal 102 can both be tablet computers, etc.

[0098] The call opposite terminal device is a device of the other party of the call, and the call opposite terminal device can be a mobile phone, a tablet computer, a wearable electronic device, a notebook computer, a smart television, or a device with a telephone function, and the embodiments of the present application do not make any limitation in this regard.

[0099] Please refer to Figure 3 A structural schematic diagram of a terminal 200 is provided for the embodiments of the present application, and the terminal 200 can be the first terminal 101 in the multi-screen cooperation call system 100 described above, for example, a mobile phone; or the terminal 200 can be the second terminal 102 in the multi-screen cooperation call system 100 described above, for example, a tablet computer. The hardware structure of the terminal 200 is described below by taking a mobile phone as an example.

[0100] As shown in FIG. 2, the terminal 200 can include a processor 201, a memory 202, a power supply 203, a display 204, a communication interface 205, and a touch screen 206. Figure 3As shown, the terminal 200 can include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a key 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc. The sensor module 280 can include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, and a rotation shaft sensor 280N, etc.

[0101] It can be understood that the structure shown in the embodiment does not constitute a specific limitation on the terminal 200. In other embodiments, the terminal 200 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0102] The processor 210 can include one or more processing units, for example: the processor 210 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0103] The processor 210 can also include memory that stores instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The cache memory can hold instructions and data that the processor 210 has recently accessed or is likely to access again. If the processor 210 needs to access the instructions or data again, it can be accessed directly from the cache memory. This avoids repeated accesses to the main memory, reducing the latency of the processor 210 and improving the efficiency of the system.

[0104] The charging management module 240 is configured to receive charging input from a charger. The power management module 241 is configured to connect the battery 242 to the charging management module 240. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the external memory, the display 294, the camera 293, and the wireless communication module 260.

[0105] The wireless communication functions of the terminal 200 can be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, and the baseband processor.

[0106] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the terminal 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0107] The mobile communication module 250 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the terminal 200. The mobile communication module 250 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1 and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 250 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves radiated by the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 250 can be arranged in the processor 210. In some embodiments, at least part of the functional modules of the mobile communication module 250 and at least part of the modules of the processor 210 can be arranged in the same device.

[0108] The wireless communication module 260 can provide a solution for wireless communication, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc., which are applied to the terminal 200. The wireless communication module 260 can be one or more devices that integrate at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 210. The wireless communication module 260 can also receive signals to be transmitted from the processor 210, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0109] In some embodiments, antenna 1 and mobile communication module 250 of terminal 200 are coupled, and antenna 2 and wireless communication module 260 are coupled, so that terminal 200 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0110] Terminal 200 implements display functions through a GPU, display screen 294, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 can include one or more GPUs that execute program instructions to generate or change display information.

[0111] The display screen 294 is configured to display images, videos, and the like. For example, in the embodiments of the present application, the display screen 294 is configured to display application icons, shortcut menu options, and a delay setting window, and the like, to guide a user to set a delay time length. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), and the like. In some embodiments, the terminal 200 can include one or N display screens 294, where N is a positive integer greater than 1.

[0112] The terminal 200 can implement a photographing function through an ISP, the camera 293, a video codec, a GPU, the display screen 294, and an application processor, and the like.

[0113] The ISP is configured to process data fed back by the camera 293. For example, when taking a photo, a shutter is opened, light is transmitted to a camera photosensitive element through a lens, and the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts the electrical signal into an image visible to the naked eye. The ISP can also perform algorithm optimization on noise, brightness, and skin color of the image. The ISP can also optimize exposure, color temperature, and the like of a photographing scene. In some embodiments, the ISP can be disposed in the camera 293.

[0114] The camera 293 is configured to capture a still image or a video. An object generates an optical image through a lens and projects the optical image onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the terminal 200 can include one or N cameras 293, where N is a positive integer greater than 1.

[0115] The external memory interface 220 can be configured to connect an external memory card, such as a Micro SD card, to extend the memory capacity of the terminal 200. The external memory card can communicate with the processor 210 via the external memory interface 220 to store data. For example, music, video, and other files can be stored in the external memory card.

[0116] The internal memory 221 can be configured to store computer-executable program codes including instructions. The processor 210 can perform various functions of the terminal 200 and data processing by executing the instructions stored in the internal memory 221. The internal memory 221 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (such as a sound play function, an image play function, etc.) required by a function, etc. The data storage area can store data (such as audio data, a phonebook, etc.) created during the use of the terminal 200, etc. In addition, the internal memory 221 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0117] The terminal 200 can implement an audio function through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D, the application processor, etc. For example, music play, recording, etc.

[0118] The audio module 270 can be configured to convert digital audio information into an analog audio signal, and to convert an analog audio input into a digital audio signal. The audio module 270 can also be configured to encode and decode an audio signal. In some embodiments, the audio module 270 can be disposed in the processor 210, or some functional modules of the audio module 270 can be disposed in the processor 210.

[0119] The speaker 270A, also referred to as a "loudspeaker", can be configured to convert an audio electrical signal into a sound signal. The terminal 200 can listen to music or a hands-free call through the speaker 270A.

[0120] The receiver 270B, also referred to as a "earpiece", can be configured to convert an audio electrical signal into a sound signal. When the terminal 200 is on a call or receives a voice message, the receiver 270B can be placed close to a human ear to listen to the voice.

[0121] The microphone 270C, also referred to as a "microphone" or a "microphone", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, a user can speak into the microphone 270C by placing his mouth close to the microphone 270C to input a sound signal into the microphone 170C. The terminal 200 can be provided with at least one microphone 270C. In other embodiments, the terminal 200 can be provided with two microphones 270C, which can not only collect sound signals, but also achieve noise reduction functions. In other embodiments, the terminal 200 can also be provided with three, four or more microphones 270C to achieve sound signal collection, noise reduction, sound source identification, directional recording and other functions.

[0122] The earphone interface 270D is used to connect a wired earphone. The earphone interface 270D can be a USB interface 230, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0123] The SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to achieve contact and separation with the terminal 200. The terminal 200 can support one or N SIM card interfaces, N being a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. The same SIM card interface 295 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 can also be compatible with external storage cards. The terminal 200 interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the terminal 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 200 and cannot be separated from the terminal 200.

[0124] It should be noted that the hardware structure of the terminal 200 described above is illustrated by taking a mobile phone as an example. Those skilled in the art should understand that the hardware structure of the first terminal 101 and the second terminal 102 can be appropriately increased or decreased on the basis of the hardware structure described above, and the embodiments of the present application do not make any limitation in this regard.

[0125] The software system of the aforementioned terminal 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to exemplify the software structure of the terminal 200. Figure 3 This is a software structure block diagram of terminal 200 according to an embodiment of the present invention.

[0126] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0127] The application layer can include a series of application packages.

[0128] like Figure 3 As shown, the application package can include applications (apps) such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS. The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0129] The calling application has the following functions: to initiate a call in response to a user's dialing or answering operation (e.g., clicking the answer button on the caller ID screen); to record a call in response to a user's recording operation (e.g., clicking the record button on the call screen); and to make an emergency call in response to a user's dialing operation (e.g., selecting an emergency contact on the emergency call screen).

[0130] When a user operates the terminal 200 to answer a call, make a call, record a call, or make an emergency call, the call application responds to the user's operation and generates corresponding instructions to send to the application framework layer.

[0131] like Figure 4 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0132] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0133] The content provider stores and retrieves data and makes the data accessible to the application programs. The data can include videos, images, audios, dialed and received phone numbers, browsing history and bookmarks, phone book, etc.

[0134] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0135] The phone manager is used to provide the communication function of the terminal 200. For example, the management of the call state (including call connection, call disconnection, etc.).

[0136] The phone manager provides classes related to the call, such as the audio service class. When the user operates the terminal 200 to receive a call, or to make a call, or to record a call, or to make an emergency call, the call application program generates a corresponding instruction to the audio service class of the application program framework layer in response to the user's operation. The resource manager provides various resources for the application program, such as localized strings, icons, pictures, layout files, video files, etc.

[0137] The notification manager enables the application program to display notification information in the status bar, which can be used to convey a message of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of the download, the message reminder, etc. The notification manager can also be a notification that appears in the form of a chart or a scroll bar text in the top status bar of the system, such as a notification of an application program running in the background, and can also be a notification that appears in the form of a dialog window on the screen. For example, the text information is prompted in the status bar, a prompt sound is emitted, the device vibrates, the indicator light flashes, etc.

[0138] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0139] The core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0140] The application program layer and the application program framework layer run in the virtual machine. The virtual machine executes the java files of the application program layer and the application program framework layer into binary files. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0141] The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (e.g., OpenGL ES), audio libraries, and the like.

[0142] The surface manager is used to manage a display subsystem and provides fusion of 2D and 3D layers for a plurality of applications.

[0143] The media libraries support playback and recording of a plurality of commonly used audio, video formats, and still image files. The media libraries can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0144] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, composition, and layer processing, and the like.

[0145] The telephony manager is only a bridge to provide access to the core library for a telephony application, and specific function implementation is completed in the audio libraries. For example, the audio libraries have an audio service class (Audio Service) corresponding to an audio service class of the application framework layer.

[0146] The HAL layer is a wrapper for Linux kernel drivers, providing an interface upward and shielding implementation details of the underlying hardware.

[0147] The HAL layer can include a Wi-Fi HAL, an audio HAL (Audio HAL), and a camera HAL (Camera HAL), and the like.

[0148] The audio HAL defines standard interfaces that must be implemented by the audio service to ensure that audio hardware functions operate normally, and is responsible for associating the Audio Flinger / Audio Policy Service with hardware devices.

[0149] The kernel layer is a layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0150] The audio drivers are mainly responsible for interaction with hardware, including audio devices, such as speakers, microphones, and the like.

[0151] To make it easier for people to more clearly understand the multi-screen collaboration call method of the embodiments of the present application, the Android system introduced above is introduced first, and the phone function of the first terminal 101 in the prior art is introduced taking the first terminal 101 being a mobile phone as an example. Figure 4 The Android system introduced above is introduced first, and the phone function of the first terminal 101 in the prior art is introduced taking the first terminal 101 being a mobile phone as an example.

[0152] The telephone function of the first terminal 101 includes a call, a call recording, and an emergency call. The following describes the process of implementing the telephone function of the first terminal 101 with reference to the accompanying drawings. Figure 4

[0153] In one implementation, as shown in FIG. 1, the process of the first terminal 101 making a call with a call opposite device is as follows. Figure 4

[0154] When the user answers an incoming call, the call application generates an open call instruction in response to the user's operation of answering the incoming call (for example, clicking the answer key of the incoming call display interface, etc.) on the first terminal 101, and calls the audio service class of the audio service of the phone manager and the audio service class of the audio library, and then sends the open call instruction to the audio hardware management module of the audio HAL, and then sends the open call instruction to the audio processor by the audio hardware management module. After receiving the open call instruction, the audio processor receives the call downlink data sent by the call opposite device and sends it to the loudspeaker for playing; and acquires the call uplink data collected by the microphone and sends it to the call opposite device.

[0155] When the user makes a call, the call application generates an open call instruction in response to the user's operation of making a call (for example, entering the phone of the main interface, selecting a related contact, and clicking to dial, etc.) on the first terminal 101, and after the call opposite device is connected, the call application generates an open call instruction. Then, the call application calls the audio service class of the audio service of the phone manager and the audio service class of the audio library, and then sends the open call instruction to the audio hardware management module of the audio HAL, and then sends the open call instruction to the audio processor by the audio hardware management module. After receiving the open call instruction, the audio processor receives the call downlink data sent by the call opposite device and sends it to the loudspeaker for playing; and acquires the call uplink data collected by the microphone and sends it to the call opposite device.

[0156] It should be noted that in some scenarios, the audio processor can also send the call downlink data to the receiver for playing, or send it to the wired earphone through the earphone interface for playing, or send it to the wireless earphone through Bluetooth for playing. At the same time, the call uplink data acquired by the audio processor can also be collected by the wireless earphone or the wired earphone connected with the terminal 200. The embodiments of the present application do not make any limitation on this.

[0157] The audio processor, that is, the Audio DSP, is a digital signal processor (Digital Signal Process, DSP) for processing audio. The Audio DSP is responsible for processing audio signals and interacting with the Audio DSP of the call opposite device.

[0158] In another implementation, as shown in FIG. 1, the process of the first terminal 101 making a call with a call opposite device is as follows.​​Figure 5 As shown, the first terminal 101 performs the process of call recording as follows:

[0159] In the process of the first terminal 101 performing a call with a call opposite terminal device, when a user performs call recording, the call application generates an opening recording instruction in response to the user's opening recording operation (for example, clicking a recording key of a call interface), and calls an audio service class in a phone manager and an audio service class in an audio library, and sends the opening recording instruction to an audio hardware management module of an audio HAL.

[0160] After the audio hardware management module receives the opening recording instruction, the first PCM device is opened, and the audio scene is specified as the uplink and downlink. Then, the first PCM device obtains the call uplink data and the call downlink data from the audio processor, and mixes the call uplink data and the call downlink data to obtain recording data. The audio hardware management module reads the recording data from the first PCM device, and sends the recording data to the call application through the audio hardware management module and the audio service class, and the call application stores the recording data.

[0161] The PCM device is a PCM device file. In a Linux system, all devices are finally abstracted into one or more device files accessible by a user space, and a process in the user space controls hardware by reading and writing the device files. For playing sound or recording sound, the audio HAL is implemented by reading and writing the PCM device file.

[0162] In the Android system, call recording needs to obtain recording data from a specified PCM device, and the recording data includes the voices of both parties of the call. The PCM device can be specified by selecting an audio scene (i.e., a use case), and one PCM device can correspond to multiple use cases.

[0163] The first PCM device, i.e., the incall record PCM, is a PCM device used to implement call recording. The incall record PCM corresponds to three use cases, which are: the uplink (usecase_incall_rec_downlink), the downlink (usecase_incall_rec_uplink), and the uplink and the downlink (usecase_incall_rec_uplink_and_downlink). By selecting the three use cases, the first PCM device can obtain the call downlink data, the call uplink data, and the call uplink data and the call downlink data, respectively.

[0164] Since the call recording needs the call uplink data and the call downlink data, the usecase_incall_rec_uplink_and_downlink needs to be selected to obtain the call uplink data and the call downlink data in the call recording scenario. That is, the first PCM device is specified by selecting the usecase_incall_rec_uplink_and_downlink, so as to realize the call recording.

[0165] In yet another implementation manner, as shown in FIG. 10, the first terminal 101 performs the emergency call in the following manner: Figure 1

[0166] When the user dials an emergency call, the call application writes the emergency call data into the audio service class of the phone manager in response to the user's operation of dialing an emergency call (for example, clicking the emergency call on the lock screen interface, clicking the personal emergency information, selecting the emergency contact dialing, etc.), the emergency call data is a piece of fixed audio data, and then the audio service class of the phone manager writes the emergency call data into the audio service class of the audio library, and then the audio service class of the audio library writes the emergency call data into the audio hardware management module of the audio HAL. Then, the audio hardware management module writes the emergency call data into the second PCM device, and the audio processor reads the emergency call data from the second PCM device and sends it to the call opposite terminal device.

[0167] If the user's operation of dialing an emergency call is to click the emergency call on the lock screen interface, click the personal emergency information, and select the emergency contact dialing, the call opposite terminal device is the device of the emergency contact. After receiving the emergency call data, the call opposite terminal device plays the emergency call data, so that the emergency contact can dial a help-seeking call through his own device for the user.

[0168] The second PCM device, that is, the incall music PCM, is a PCM device used to realize the emergency call function. The incall music PCM also has a corresponding usecase, which is different from the usecase corresponding to the incall record PCM. For example, the usecase corresponding to the incall music PCM can be USECASE_INCALL_MUSIC_UPLINK.

[0169] From the above content, it can be known that in the existing call process, the audio processor receives the call downlink data and sends it to the loudspeaker for playing, the Android system does not provide an interface for directly obtaining the call downlink data, and the audio system cannot directly obtain the call downlink data from the audio processor. However, in the existing call recording process, the first PCM device can obtain the call uplink data and the call downlink data from the audio processor.

[0170] ​Therefore, the multi-screen cooperative call method provided by the embodiments of the present application improves the telephone function of the first terminal 101 in the prior art, so that the first terminal 101 receives the call downlink data of the call opposite terminal device, and sends the call downlink data to the second terminal 102 for playing, and receives the first call uplink data collected by the second terminal 102, and sends the first call uplink data to the call opposite terminal device. The following will be described in detail.

[0171] The following will take the first terminal 101 as a mobile phone and the second terminal 102 as a tablet computer as an example to describe the multi-screen cooperative call method of the embodiments of the present application in detail.

[0172] Please refer to Figure 6 , a flowchart of a multi-screen cooperative call method provided by the embodiments of the present application. The multi-screen cooperative call method is applied to Figure 3 The multi-screen cooperative call method can include S301-S304.

[0173] S301, in the case that the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, the first terminal receives the incoming call of the call opposite terminal device.

[0174] S302, in response to the user's call answering operation on the interface of the first terminal in the second terminal, the first terminal receives the call downlink data sent by the call opposite terminal device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data.

[0175] S303, the second terminal collects the first call uplink data, and sends the first call uplink data to the call opposite terminal device through the first terminal.

[0176] The first terminal 101 is connected with the second terminal 102 and the second terminal 102 displays the interface of the first terminal, that is, the first terminal 101 and the second terminal 102 are in multi-screen cooperation. The interface of the first terminal refers to the window of the first terminal 101 displayed on the second terminal 102 in mirror image.

[0177] When the first terminal 101 and the second terminal 102 are in multi-screen cooperation, if the call opposite terminal device calls, the first terminal 101 displays the incoming call display interface, and at the same time, the interface of the first terminal in the second terminal 102 synchronously displays the incoming call display interface.

[0178] At this time, if the user operates the incoming call display interface displayed on the second terminal 102 to answer the call, it can be considered that the user tends to call the other party through the second terminal 102, so the call sound is switched from the first terminal 101 to the second terminal 102.

[0179] That is, the first terminal 101 receives the call downlink data sent by the call opposite terminal device and sends it to the second terminal 102, and the first terminal 101 receives the first call uplink data collected by the second terminal 102 and sends it to the call opposite terminal device. The call downlink data refers to the voice of the other party of the call. The first call uplink data refers to the voice of the user collected by the sound collecting device (for example, a microphone) of the second terminal 102.

[0180] That is, the first terminal 101 receives the voice of the other party of the call sent by the call opposite terminal device and sends it to the second terminal 102, and the sound playing device (for example, a loudspeaker) of the second terminal 102 plays the voice of the other party of the call. At the same time, the sound collecting device (for example, a microphone) of the second terminal 102 collects the voice of the user and sends it to the call opposite terminal device through the first terminal 101.

[0181] For example, please refer to Figure 7 The mobile phone and the tablet computer are in multi-screen cooperation, that is, the mobile phone is connected with the tablet computer, and the tablet computer displays the interface 401 of the mobile phone. The mobile phone has an incoming call, and the interface 1021 of the mobile phone in the tablet computer displays the incoming call display interface. If the user clicks the answer key in the incoming call display interface of the tablet computer to answer the call, the voice of the other party of the call is played from the tablet computer, and at the same time, the tablet computer collects the voice of the user and sends it to the other party of the call through the mobile phone.

[0182] S304, in response to the user's starting recording operation, the first terminal obtains recording data, wherein the recording data includes call downlink data and first call uplink data.

[0183] The starting recording operation can be that the user clicks the recording icon in the call interface of the first terminal 101, or that the user clicks the recording icon in the call interface synchronously displayed by the first terminal interface of the second terminal 102.

[0184] That is, in the case of multi-screen cooperation between the first terminal 101 and the second terminal 102, if the first terminal 101 starts recording after starting the virtual call, the recording data includes the call downlink data and the first call uplink data. The virtual call refers to that the first terminal 101 is in the call state with the call opposite terminal device, but the call voice is played from the second terminal 102, that is, the user has a conversation with the other party of the call through the second terminal 102. Starting the virtual call means switching the call voice from the first terminal 101 to the second terminal 102.

[0185] The above describes one application scenario of starting the virtual call by the first terminal 101. In addition to the above application scenario, the first terminal 101 can also start the virtual call in some other scenarios, which will be described in combination with examples.

[0186] In one possible implementation manner, in Figure 3Based on this, the multi-screen collaborative call method provided in the embodiments of this application may further include S301-1, S302-1, S303 and S304.

[0187] S301-1, when the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, in response to the user's dialing operation on the interface of the first terminal in the second terminal, the first terminal calls the other end of the call.

[0188] S302-1 After the other end of the call is connected, the first terminal receives the downlink data sent by the other end of the call and sends it to the second terminal, and the second terminal plays the downlink data.

[0189] S303, the second terminal collects the uplink data of the first call and sends the uplink data of the first call to the device at the other end of the call through the first terminal.

[0190] S304, in response to the user's operation to start recording, the first terminal obtains recording data, wherein the recording data includes downlink data of the call and uplink data of the first call.

[0191] When the first terminal 101 and the second terminal 102 are in multi-screen collaboration, if the user makes a call by operating the first terminal interface displayed on the second terminal 102, it can be assumed that the user prefers to make a call through the second terminal 102. Therefore, the first terminal 101 starts a virtual call and switches the call audio from the first terminal 101 to the second terminal 102.

[0192] For example, please refer to Figure 8 The phone and tablet are in multi-screen collaboration, meaning the phone and tablet are connected, and the tablet displays the phone's interface 401. If the user uses the phone interface 401 on the tablet to make a call—for example, by clicking the phone icon on the phone interface 401, entering the phone app, selecting the relevant contact, and pressing the dial button—the other party will be contacted. After the other party answers, their voice will be emitted from the tablet, and simultaneously, the tablet will capture the user's voice and transmit it to the other party via the phone.

[0193] In another possible implementation, Figure 3 Based on this, the multi-screen collaborative call method provided in the embodiments of this application may further include S301-2, S302-2, S303 and S304.

[0194] S301-2, when the first terminal is in a call with the other end of the call, the first terminal displays a pop-up window, which is used to prompt the user to perform multi-screen collaboration between the first terminal and the second terminal.

[0195] S302-2, in response to the operation of the user on the connection icon in the pop-up box, the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, the first terminal receives the call downlink data sent by the call opposite terminal device and sends to the second terminal, and the second terminal plays the call downlink data.

[0196] S303, the second terminal collects the first call uplink data, and sends the first call uplink data to the call opposite terminal device through the first terminal.

[0197] S304, in response to the opening recording operation of the user, the first terminal obtains the recording data, wherein the recording data includes the call downlink data and the first call uplink data.

[0198] When the user is calling the call opposite terminal device through the first terminal 101, if the user operates the first terminal 101 and the second terminal 102 to make the first terminal 101 and the second terminal 102 multi-screen cooperation, it can be considered that the user tends to call the other party through the second terminal 102, so the first terminal 101 opens the virtual call and switches the call sound from the first terminal 101 to the second terminal 102.

[0199] For example, please refer to Figure 9 , the user is calling the other party through the mobile phone, and the pop-up box 402 is displayed in the mobile phone to prompt the user to connect the mobile phone with the tablet computer. If the user clicks the connection icon in the pop-up box 402, the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, that is, the mobile phone is connected with the tablet computer, and the multi-screen cooperation is started. At this time, the sound of the other party is emitted from the tablet computer, and at the same time, the tablet computer collects the sound of the user and sends it to the other party through the mobile phone, so that the user can call the other party through the tablet computer. In another possible implementation, on the basis of Figure 3 , the multi-screen cooperation call method provided by the embodiment of the application can further include S301-3, S302-3, S303 and S304.

[0200] S301-3, in the case that the first terminal is calling the call opposite terminal device and the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, the first terminal displays the first notification panel.

[0201] S302-3, in response to the operation of the user on the audio / video switching icon in the first notification panel in the first terminal, the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, the first terminal receives the call downlink data sent by the call opposite terminal device and sends to the second terminal, and the second terminal plays the call downlink data.

[0202] S303, the second terminal collects first call uplink data, and sends the first call uplink data to the call opposite end device through the first terminal.

[0203] S304, in response to the user's opening recording operation, the first terminal obtains recording data, wherein the recording data comprises call downlink data and first call uplink data.

[0204] When the user is calling with the call opposite end device through the first terminal 101 and the first terminal 101 is in multi-screen cooperation with the second terminal 102, if the user operates the first terminal 101 to switch the audio and video to the second terminal, the first terminal 101 opens a virtual call and switches the call sound from the first terminal 101 to the second terminal 102.

[0205] For example, please refer to Figure 10 , the user is calling with the other party through the mobile phone, and the mobile phone is in multi-screen cooperation with the tablet computer, the user slides down the status bar of the mobile phone to open the first notification panel 403, if the user clicks the "audio and video switch to tablet" icon in the first notification panel 403, the call sound will be switched from the mobile phone to the tablet computer, that is, the sound of the other party is emitted from the tablet computer, at the same time, the tablet computer collects the user's voice and sends it to the other party through the mobile phone, and the user calls with the other party through the tablet computer.

[0206] In another possible implementation, on the basis of Figure 4 , the multi-screen cooperation call method provided by the embodiment of the application can further comprise S301-4, S302-4, S303 and S304.

[0207] S301-4, in the case that the first terminal is calling with the call opposite end device and the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, the second terminal displays the first notification panel.

[0208] S302-4, in response to the user's operation on the audio and video switching icon in the first notification panel in the second terminal, the first terminal is connected with the second terminal and the second terminal displays the interface of the first terminal, the first terminal receives the call downlink data sent by the call opposite end device and sends it to the second terminal, and the second terminal plays the call downlink data.

[0209] S303, the second terminal collects first call uplink data, and sends the first call uplink data to the call opposite end device through the first terminal.

[0210] S304, in response to the user's opening recording operation, the first terminal obtains recording data, wherein the recording data comprises call downlink data and first call uplink data.

[0211] When a user makes a call with the other end of the call through the first terminal 101, or when the first terminal 101 and the second terminal 102 are in multi-screen collaboration, if the user operates the second terminal 102 to switch the audio and video to the second terminal 102, the first terminal 101 will start a virtual call and switch the call audio from the first terminal 101 to the second terminal 102.

[0212] For example, please refer to Figure 4 The user is having a call with another party via their mobile phone, and the mobile phone and tablet are in multi-screen collaboration. The user swipes down the status bar of the tablet to open the first notification panel 403. If the user clicks the "Switch audio / video to tablet" icon in the first notification panel 403, the call audio will switch from the mobile phone to the tablet. That is, the other party's voice will be emitted from the tablet. At the same time, the tablet will capture the user's voice and send it to the other party via the mobile phone. The user can then have a call with the other party via the tablet.

[0213] Of course, in addition to the possible implementations described above, those skilled in the art can design other implementations to initiate virtual calls, and this application embodiment does not impose any restrictions on this.

[0214] The above describes the complete call process in a virtual call scenario. However, from... Figure 4 As described in the existing telephone function implementation, the Android system does not provide an interface for directly obtaining downlink call data. Furthermore, the audio processor directly obtains uplink call data collected by the microphone of the first terminal 101 and sends it to the other end of the call. However, the first PCM device can obtain both uplink and downlink call data from the audio processor, and the audio processor can read emergency call data from the second PCM device.

[0215] Therefore, to realize the virtual call function in the embodiments of this application, that is, the first terminal 101 receives the downlink call data of the call peer device and sends it to the second terminal 102 for playback, and receives the first uplink call data collected by the second terminal 102 and sends it to the call peer device, the functions of the first PCM device and the second PCM device can be borrowed.

[0216] from Figure 4 As can be seen from the introduction, in a call recording scenario, the first PCM device can obtain the call uplink data and call downlink data from the audio processor. Therefore, in a virtual call scenario, the first PCM device can obtain the call downlink data from the audio processor.

[0217] Meanwhile, since call recording may also be required in virtual call scenarios, the first PCM device is configured to include uplink and downlink paths, that is, usecase_incall_rec_uplink_and_downlink is selected to obtain call uplink and downlink data.

[0218] In addition, from Figure 11 As described above, in an emergency call scenario, the audio hardware management module can write emergency call data to the second PCM device, and the audio processor reads the emergency call data from the second PCM device and sends it to the other end of the call. Therefore, the first uplink data sent by the second terminal 102 can be written to the second PCM device, and then the audio processor reads the first uplink data from the second PCM device and sends it to the other end of the call.

[0219] Therefore, the audio processor contains both downlink data sent by the other end of the call and uplink data read from the second PCM device.

[0220] However, in a virtual call scenario, the first PCM device includes both uplink and downlink paths. Therefore, the first PCM device can obtain both downlink and uplink data from the audio processor, inevitably causing an echo when played on the second terminal 102. Therefore, it is necessary to disable the uplink path of the first PCM device to prevent it from obtaining the uplink data from the audio processor.

[0221] Meanwhile, since the user communicates with the other party through the second terminal 102 in a virtual call scenario, the first terminal 101 needs to disable its own audio device when initiating the virtual call to avoid echoes. The audio device can include a playback device and a reception device. The playback device can be a speaker, receiver, wired headphones, wireless headphones, etc., and the reception device can be a microphone, wired headphones, wireless headphones, etc. This application embodiment does not impose any limitations on this; the following embodiments use a speaker and a microphone as examples.

[0222] Based on the above analysis, compared to Figure 3 ,like Figure 11 As shown, to implement virtual calls, a collaboration module, a virtual call hardware management module, a transmission module, and a virtual call buffer can be added to the first terminal 101. Correspondingly, a transmission module can be added to the second terminal 102. The virtual call hardware management module is... Figure 6 The audio in HAL is shown.

[0223] The collaboration module is activated when the first terminal 101 and the second terminal 102 are connected in a multi-screen collaboration mode. It is mainly responsible for monitoring the call status and controlling the call logic. For example, when it detects that a virtual call has been started, it sends an instruction to start a virtual call to the virtual call hardware management module; when it detects that a virtual call has been stopped, it sends an instruction to stop a virtual call to the virtual call hardware management module.

[0224] It should be noted that the second terminal 102 also has a corresponding collaboration module. Figure 7 (Not shown in the image), the collaboration module in the second terminal 102 can monitor the user's operations on the tablet and generate corresponding instructions. For example, if it detects... Figure 10 or Figure 11 The user operation shown generates a call request which is sent to the coordination module of the first terminal 101. For example, monitoring... Figure 12 The user operation shown generates a call switching request and sends it to the coordination module of the first terminal 101.

[0225] The transmission module can be represented as virModem transmission, which is mainly responsible for audio data transmission between the first terminal 101 and the second terminal 102.

[0226] The virtual call hardware management module, which can be represented as virModem hal, is mainly used to: control the opening and closing of the PCM device, set the audio scene, read and write audio data, and mix the downlink data of the call and the uplink data of the first call when it receives the virtual call start command or virtual call stop command sent by the coordination module.

[0227] The virtual call buffer is used to store the data after the virtual call hardware management module mixes the downlink data of the call and the uplink data of the first call.

[0228] The following is combined Figure 12 and Figure 6 to Figure 10 This paper provides a detailed introduction to the implementation of calls and call recording in virtual call scenarios.

[0229] Please refer to Figure 11 The processes described above, including S302, S302-1, S302-2, S302-3, and S302-3, where the first terminal receives downlink call data from the peer device and sends it to the second terminal, and the second terminal plays the downlink call data, and S303, where the second terminal collects the first uplink call data and sends the first uplink call data to the peer device through the first terminal, may include:

[0230] S11, in response to the user's virtual call initiation operation, the call application sends a virtual call initiation command to the collaboration module. The virtual call initiation operation can be...Figure 12 Any of the following can be used to switch the call audio from a mobile phone to a tablet.

[0231] S12, the collaboration module will send the virtual call start command to the virtual call hardware management module.

[0232] S13, in response to receiving the virtual call start command, the virtual call hardware management module disables the audio device of the first terminal, turns on the first PCM device, and prevents the first PCM device from acquiring uplink data.

[0233] When switching the call audio from a mobile phone to a tablet, in order to ensure that the call audio is not played or picked up by the phone's speaker and microphone, the virtual call hardware management module needs to disable the phone's speaker and microphone. This prevents the phone's speaker from playing the other party's voice and the phone's microphone from picking up the user's voice.

[0234] S15-1, The audio processor receives downlink data from the other end of the call;

[0235] S15-2, The first PCM device obtains downlink call data from the audio processor;

[0236] S15-3, The virtual call hardware management module reads the downlink call data from the first PCM device and sends the downlink call data to the transmission module of the first terminal;

[0237] S15-4, The transmission module of the first terminal sends the downlink data of the call to the transmission module of the second terminal;

[0238] S15-5, the transmission module of the second terminal sends the downlink data of the call to the audio device of the second terminal;

[0239] S15-6, The audio device of the second terminal plays downlink data of the call.

[0240] S15-1 to S15-6 are the implementation process for the tablet computer to play the voice of the other party in the call.

[0241] S16-1, The audio device of the second terminal collects the uplink data of the first call and transmits the uplink data of the first call to the transmission module of the second terminal;

[0242] S16-2, the transmission module of the second terminal sends the first call uplink data to the transmission module of the first terminal;

[0243] S16-3, the transmission module of the first terminal sends the first call uplink data to the virtual call hardware management module;

[0244] S16-4, the virtual call hardware management module writes the first call uplink data into the second PCM device;

[0245] S16-5, the audio processor reads the first call uplink data from the second PCM device and sends it to the call opposite end device.

[0246] S16-1-S16-5, namely, the implementation process of collecting the user's voice by the tablet computer and sending it to the call opposite end device through the mobile phone.

[0247] The interaction process between the above-mentioned various modules is the call process in the virtual call scenario. Since the call process in the virtual call scenario needs to use the first PCM device, and the virtual call hardware management module prohibits the first PCM device from obtaining uplink data from the audio processor, the audio hardware management module will not be able to read the recording data from the first PCM device. Therefore, in order to realize the call recording in the virtual call scenario, the above-mentioned call process also needs to include:

[0248] S17-1, the virtual call hardware management module mixes the call downlink data and the first call uplink data;

[0249] S17-2, the virtual call hardware management module stores the mixed data into the virtual call cache area.

[0250] As shown in Figure 5 , the virtual call hardware management module mixes the call downlink data and the first call uplink data, and stores the mixed data, i.e. recording data, into the virtual call cache area. The audio hardware management module can read the recording data from the virtual call cache area to realize the call recording in the virtual call scenario.

[0251] Therefore, please continue to refer to Figure 13 , the process of obtaining the recording data by the first terminal in response to the user's start recording operation in the foregoing S304 can include:

[0252] S21, the call application program sends a start recording instruction to the audio service class in response to the start recording operation;

[0253] S22, the audio service class sends the start recording instruction to the audio hardware management module;

[0254] S17-3, in response to receiving the start recording instruction, the audio hardware management module obtains the recording data from the virtual call cache area;

[0255] S17-4, the audio hardware management module sends the recording data to the audio service class;

[0256] S17-5, the audio service class sends the recording data to the communication application program.

[0257] S17-6, the communication application stores the recording data.

[0258] In some scenarios, the first terminal 101 can also close the virtual call to switch the call sound from the second terminal 102 back to the first terminal 101.

[0259] Therefore, on the basis of Figure 14 , please refer to Figure 13 After S304, the call method of multi-screen cooperation provided by the embodiment of the application can further include S305-S306.

[0260] S305, in the case that the first terminal is connected with the second terminal, the second terminal displays the interface of the first terminal, and the user makes a call with the call opposite terminal device through the second terminal, the first terminal plays the call downlink data and collects the second call uplink data to send to the call opposite terminal device in response to the user's closing operation on the interface of the first terminal in the second terminal.

[0261] When the first terminal 101 and the second terminal 102 are in multi-screen cooperation, and the user makes a call with the call opposite terminal device through the second terminal 102, if the user operates the second terminal 102 to disconnect the connection with the first terminal 101, the first terminal 101 closes the virtual call to switch the call sound from the second terminal 102 back to the first terminal 101.

[0262] That is, the first terminal 101 receives the call downlink data sent by the call opposite terminal device and plays it, and collects the second call uplink data to send to the call opposite terminal device. The second call uplink data refers to the user's voice collected by the sound collecting device (for example, a microphone) of the first terminal 101.

[0263] That is, the first terminal 101 receives the voice of the other party of the call sent by the call opposite terminal device and plays it through the sound playing device (for example, a loudspeaker), and collects the user's voice through the sound collecting device (for example, a microphone) and sends it to the call opposite terminal device.

[0264] For example, please refer to Figure 15 , the user is making a call with the other party of the call through the tablet computer, and the mobile phone and the tablet computer are in multi-screen cooperation, that is, the mobile phone and the tablet computer are connected and the tablet computer displays the interface 401 of the mobile phone. The user first clicks the multi-screen cooperation disconnection icon in the interface 401 of the mobile phone of the tablet computer, the tablet computer displays a pop-up box 404 to prompt the user to confirm whether to disconnect the connection between the mobile phone and the tablet computer, and the user clicks the disconnection icon in the pop-up box 404, then the first terminal and the second terminal are disconnected and stop multi-screen cooperation. At this time, the voice of the other party of the call is played from the mobile phone, and the mobile phone collects the user's voice and sends it to the other party of the call, and the user makes a call with the other party of the call through the mobile phone.

[0265] S306, the first terminal obtains the recording data, wherein the recording data comprises the call downlink data and the second call uplink data.

[0266] After the first terminal 101 closes the virtual call, the call sound is switched back from the second terminal 102 to the first terminal 101, and the user carries on a conversation with the other party of the call through the second terminal 102. At this time, it is the call flow of the first terminal 101 in the prior art. In this case, if the first terminal 101 still records the call, the recording data comprises the call downlink data and the second call uplink data.

[0267] When the first terminal 101 and the second terminal 102 are in multi-screen cooperation, and the user carries on a call with the other party of the call through the second terminal 102, if the user operates the second terminal 102 to disconnect the connection with the first terminal 101, the first terminal 101 closes the virtual call and switches the call sound back from the second terminal 102 to the first terminal 101.

[0268] The above describes one application scenario of closing the virtual call by the first terminal 101. In addition to the above application scenario, the first terminal 101 can also close the virtual call in some other scenarios, which will be described below with examples.

[0269] In a possible implementation, based on Figure 13 after S304, the multi-screen cooperation call method provided by the embodiment of the application can further comprise S305-1-S306.

[0270] S305-1, in the case that the first terminal is connected with the second terminal, the second terminal displays the interface of the first terminal, and the user carries on a call with the other party of the call through the second terminal, the first terminal displays a second notification panel;

[0271] In response to the operation of the user on the disconnection icon in the second notification panel in the first terminal, the first terminal plays the call downlink data and collects the second call uplink data to send to the other party of the call.

[0272] S306, the first terminal obtains the recording data, wherein the recording data comprises the call downlink data and the second call uplink data.

[0273] When the first terminal 101 and the second terminal 102 are in multi-screen cooperation, and the user carries on a call with the other party of the call through the second terminal 102, if the user operates the first terminal 101 to disconnect the connection with the second terminal 102, the first terminal 101 closes the virtual call and switches the call sound back from the second terminal 102 to the first terminal 101.

[0274] For example, refer to Figure 16For example, please refer to Figure 17 , the user is talking with the other party through the tablet, and the phone and the tablet are in multi-screen cooperation, the user slides down the status bar of the phone to open the second notification panel 405, if the user clicks the "disconnect" icon in the second notification panel 405, the call sound will be switched back from the tablet to the phone, that is, the voice of the other party is emitted from the phone, at the same time, the phone collects the user's voice and sends it to the other party, and the user talks with the other party through the phone.

[0275] In another possible implementation, on the basis of Figure 13 , after S304, the multi-screen cooperation call method provided by the embodiment of the application can further include S305-2-S306.

[0276] S305-2, in the case that the first terminal is connected with the second terminal, the second terminal displays the interface of the first terminal, and the user is talking with the other party through the second terminal, the first terminal displays the second notification panel;

[0277] In response to the user's operation on the audio / video switching icon in the second notification panel in the first terminal, the first terminal plays the call downlink data and collects the second call uplink data to send to the other party.

[0278] S306, the first terminal obtains the recording data, wherein the recording data includes the call downlink data and the second call uplink data.

[0279] In the case that the first terminal 101 and the second terminal 102 are in multi-screen cooperation, and the user is talking with the other party through the second terminal 102, if the user operates the first terminal 101 to switch the audio / video back to the first terminal 101, the first terminal 101 closes the virtual call and switches the call sound from the second terminal 102 back to the first terminal 101.

[0280] For example, please refer to Figure 17 , the user is talking with the other party through the tablet, and the phone and the tablet are in multi-screen cooperation, the user slides down the status bar of the phone to open the second notification panel 405, if the user clicks the "disconnect" icon in the second notification panel 405, the call sound will be switched back from the tablet to the phone, that is, the voice of the other party is emitted from the phone, at the same time, the phone collects the user's voice and sends it to the other party, and the user talks with the other party through the phone.

[0281] In another possible implementation, on the basis of Figure 5 , after S304, the multi-screen cooperation call method provided by the embodiment of the application can further include S305-3-S306.

[0282] S305-3, in the case that the first terminal is connected with the second terminal, the second terminal displays the interface of the first terminal, and the user makes a call with a call opposite terminal device through the second terminal, the second terminal displays a second notification panel;

[0283] In response to the user's operation on the audio / video switching icon in the second notification panel in the second terminal, the first terminal plays the call downlink data and collects the second call uplink data to send to the call opposite terminal device.

[0284] S306, the first terminal obtains the recording data, wherein the recording data comprises the call downlink data and the second call uplink data.

[0285] In the case that the first terminal 101 and the second terminal 102 are in multi-screen cooperation, and the user makes a call with a call opposite terminal device through the second terminal 102, if the user operates the second terminal 102 to switch the audio / video back to the first terminal 101, the first terminal 101 closes the virtual call and switches the call sound from the second terminal 102 back to the first terminal 101.

[0286] For example, please refer to Figure 18 , the user is making a call with the other party through the tablet computer, and the mobile phone and the tablet computer are in multi-screen cooperation, the user slides down the status bar of the tablet computer to open the second notification panel 405, if the user clicks the "audio / video switching to mobile phone" icon in the second notification panel 405, the call sound will be switched back from the tablet computer to the mobile phone, that is, the voice of the other party of the call is emitted from the mobile phone, at the same time, the mobile phone collects the user's voice to send to the other party of the call, and the user makes a call with the other party of the call through the mobile phone.

[0287] Of course, in addition to the above-mentioned possible implementation manners, those skilled in the art can also design other implementation manners to close the virtual call, for example, if the first terminal 101 detects that the first terminal 101 is close to the human ear through the built-in proximity sensor, the virtual call is closed, and the present application embodiment does not make any limitation on this. It should be pointed out that the above-mentioned on the basis of Figure 19 , the application scenarios of closing the virtual call of the first terminal 101 are introduced. Those skilled in the art should understand that the above-mentioned S305-S306, or S305-1-S306, or S305-2-S306, or S305-3-S306, can also be applied after S304 in the above-mentioned S301-1-S304, or S301-2-S304, or S301-3-S304, or S304-4-S304, or S301-4-S304, and the present application embodiment does not make any limitation on this.

[0288] After the first terminal 101 closes the virtual call, the call sound needs to be played and collected from the speaker and microphone of the first terminal 101, therefore, the virtual call hardware management module needs to release the disabling of the speaker and microphone of the first terminal 101. Meanwhile, since the virtual call hardware management module prohibits the first PCM device from obtaining the uplink data from the audio processor in the virtual call scenario, after the first terminal 101 closes the virtual call, the virtual call hardware management module also needs to cancel the prohibition of the first PCM device from obtaining the uplink data from the audio processor.

[0289] The detailed implementation of the first terminal 101 closing the virtual call, and the call and call recording after closing the virtual call will be introduced below in combination with Figure 18 and Figure 14 to Figure 17 .

[0290] Please refer to Figure 19 , the process of the first terminal playing the call downlink data and collecting the second call uplink data to send to the call opposite end device in S305, S305-1, S302-2, S302-3 and S302-3 introduced above can include:

[0291] S31, the call application program sends a closing virtual call instruction to the cooperation module in response to the user's closing virtual call operation.

[0292] The closing virtual call operation can be any one of Figure 19 , which is used to switch the call sound from the tablet computer back to the mobile phone.

[0293] S32, the cooperation module sends the closing virtual call instruction to the virtual call hardware management module.

[0294] S33, in response to receiving the closing virtual call instruction, the virtual call hardware management module releases the disabling of the audio device of the first terminal, and cancels the prohibition of the first PCM device from obtaining the uplink data.

[0295] S35-1, the audio device of the first terminal collects the second call uplink data and transmits the second call uplink data to the audio processor;

[0296] S35-2, the audio processor sends the second call uplink data to the call opposite end device.

[0297] S36-1, the audio processor receives the call downlink data sent by the call opposite end device and transmits it to the audio device of the first terminal;

[0298] S36-2, the audio device of the first terminal plays the call downlink data.

[0299] Please refer to Figure 18, the speaker of the first terminal 101 plays the voice of the other party of the call, i.e., the call downlink data sent by the call opposite device, and the microphone of the first terminal 101 collects the voice of the user, i.e., the second call uplink data.

[0300] The interaction process between the above-mentioned various modules is the call flow after the first terminal 101 closes the virtual call. The call recording flow after the first terminal 101 closes the virtual call further includes, on the basis of the above-mentioned call flow:

[0301] S37-1, the first PCM device obtains the call downlink data and the second call uplink data from the audio processor and performs mixing.

[0302] As shown in Figure 4 , the audio hardware management module reads the mixed data of the call downlink data and the second call uplink data from the first PCM device, i.e., the recording data, so as to realize the call recording after the virtual call is closed.

[0303] Therefore, please continue to refer to Figure 11 , the process in which the first terminal obtains the recording data in the foregoing S306 can include:

[0304] S37-2, the audio hardware management module reads the recording data from the first PCM device;

[0305] S37-3, the audio hardware management module sends the recording data to the audio service class;

[0306] S37-4, the audio service class sends the recording data to the communication application program.

[0307] S37-5, the communication application program stores the recording data.

[0308] From Figure 20 the introduction of the implementation of the existing telephone function, it can be known that when the first terminal 101 performs call recording, the audio hardware management module needs to open the first PCM device and read the call uplink data and the call downlink data from the first PCM device. From Figure 11 , it can be known that in the virtual call scenario, the virtual call hardware management module prohibits the first PCM device from obtaining the uplink data from the audio processor, and the virtual call hardware management module reads the call downlink data from the first PCM device. Obviously, the call recording and the virtual call both need to use the first PCM device, and the data to be obtained is different, so there may be a conflict. Therefore, on the basis of the above-mentioned content, the process of the first terminal 101 starting or closing the recording and the process of the first terminal 101 starting or closing the virtual call are introduced below.

[0309] First, the process of starting recording by the first terminal 101 is introduced. The process of obtaining the recording data by the first terminal in response to the starting recording operation of the user in S304 can include S3041-S3045.

[0310] S3041, the call application sends a starting recording instruction to the audio service class in response to the starting recording operation.

[0311] S3042, the audio service class sends the starting recording instruction to the audio hardware management module.

[0312] In this embodiment, the audio hardware management module can maintain a recording flag bit, which has a first identifier (for example, true) and a second identifier (for example, false), wherein the first identifier represents that the first terminal 101 is currently recording a call, and the second identifier represents that the first terminal 101 is not currently recording a call.

[0313] For example, when the first terminal 101 is started, the audio hardware management module can initialize the recording flag bit to the second state. At the same time, when the first terminal 101 starts recording a call, the audio hardware management module can set the recording flag bit to the first state; and when the first terminal 101 stops recording a call, the audio hardware management module can reset the recording flag bit to the second state.

[0314] S3043, in response to receiving the starting recording instruction, the audio hardware management module determines whether the first terminal is currently recording a virtual call.

[0315] The audio hardware management module responds to the starting recording instruction and first determines whether the first terminal 101 is currently recording a virtual call. For example, the audio hardware management module determines whether the audio device of the first terminal 101 is disabled. If it is disabled, it means that the first terminal 101 is currently recording a virtual call; if it is not disabled, it means that the first terminal 101 is not currently recording a virtual call. For another example, the audio hardware management module determines whether the second PCM device is started. If it is started, it means that the first terminal 101 is currently recording a virtual call; if it is not started, it means that the first terminal 101 is not currently recording a virtual call. The present embodiment does not make any limitation in this regard.

[0316] The audio hardware management module can also determine whether the first terminal 101 is currently recording a virtual call by detecting a virtual call flag bit. In this embodiment, the virtual call hardware management module maintains a virtual call flag bit, which has a first state (for example, true) and a second state (for example, false), wherein the first state represents that the first terminal 101 is currently recording a virtual call, and the second state represents that the first terminal 101 is not currently recording a virtual call.

[0317] For example, when the first terminal 101 enables multi-screen collaboration, the virtual call hardware management module can initialize the virtual call flag and set it to the second state. Simultaneously, when the first terminal 101 enables a virtual call, the virtual call hardware management module can set the virtual call flag to the first state; and when the first terminal 101 disables a virtual call, the virtual call hardware management module can reset the virtual call flag to the second state.

[0318] Therefore, the process by which the audio hardware management module in S3043 determines whether the first terminal is currently making a virtual call can include:

[0319] The audio hardware management module detects the virtual call flag bit; if the virtual call flag bit is in the first state, the first terminal is currently making a virtual call; if the virtual call flag bit is in the second state, the first terminal is not currently making a virtual call.

[0320] S3044, if the first terminal is currently making a virtual call, the audio hardware management module obtains the recording data from the virtual call buffer. The recording data is stored by the virtual call hardware management module after mixing the downlink data of the call and the uplink data of the first call.

[0321] If the user enables call recording, the first terminal 101 is conducting a virtual call, that is, if... Figure 21 As shown, the first PCM device obtains downlink call data from the audio processor, and the audio processor reads uplink call data from the second PCM device. Since the first PCM device is in use, and the virtual call hardware management module prohibits the first PCM device from obtaining uplink data from the audio processor, the audio hardware management module will be unable to read the recording data from the first PCM device. Therefore, as... Figure 19 As shown, the virtual call hardware management module also needs to mix the downlink data of the call and the uplink data of the first call, and store the mixed data, i.e. the recording data, in the virtual call buffer. The audio hardware management module can then read the recording data from the virtual call buffer.

[0322] S3045, if the second terminal is not currently making a virtual call, the audio hardware management module opens the first PCM device and reads the recording data from the first PCM device. The recording data is obtained by the first PCM device from the audio processor by obtaining downlink data of the call and uplink data of the second call and mixing them.

[0323] If the user enables call recording while the first terminal 101 is in the middle of a call, i.e. Figure 22As shown, the audio processor receives the call downlink data sent by the call opposite end device and sends it to the speaker for playing, and the audio processor acquires the second call uplink data collected by the microphone and sends it to the call opposite end device. Since the first PCM device is not used, as shown in Figure 22 As shown, the audio hardware management module needs to open the first PCM device, and set the first PCM device to acquire the call downlink data and the second call uplink data from the audio processor and perform mixing, and the audio hardware management module reads the recording data from the first PCM device.

[0324] In order to better illustrate the above process of S3041-S3045, the following describes the process through the interactive schematic diagram as shown in Figure 12 As shown, the process of starting recording by the first terminal 101 can include: Figure 18

[0325] S21, when the user performs a start recording operation, for example, the user clicks the "recording" icon in the call interface, the call application generates a start recording instruction in response to the start recording operation, and sends the start recording instruction to the audio service class;

[0326] S22, the audio service class sends the start recording instruction to the audio hardware management module;

[0327] S23, the audio hardware management module opens the first PCM device, wherein the audio hardware management module can select usecase_incall_rec_uplink_and_downlink to specify the first PCM device;

[0328] S24, the audio hardware management module sets the recording mark position to true;

[0329] S25, the audio hardware management module detects the virtual call mark bit;

[0330] S26, if the audio hardware management module detects that the virtual call mark bit is true, the audio hardware management module acquires the recording data from the virtual call buffer area and sends it to the audio service class, and then sends it to the call application for storage by the call application;

[0331] The audio data in the virtual call buffer area is stored after the virtual call hardware management module mixes the call downlink data and the first call uplink data, and the detailed implementation can be referred to Figure 11 , which will not be repeated here.

[0332] ​S27, if the audio hardware management module detects that the virtual call flag bit is false, the audio hardware management module reads the recording data from the first PCM device and sends the recording data to the audio service class, and then sends the recording data to the call application by the audio service class, and stores the recording data by the call application.

[0333] The audio data in the first PCM device is obtained by mixing the call downlink data and the second call uplink data by the first PCM device, and the detailed implementation can be referred to Figure 20 , which will not be repeated here.

[0334] The process of closing the recording of the first terminal 101 is introduced. After S3041-S3045, the call method provided by the embodiment of the application can further include S304a-S304e.

[0335] S304a, the call application sends a close recording instruction to the audio service class in response to the user's close recording operation.

[0336] In the process of recording the call of the first terminal 101, if the user clicks the "recording" icon in the call interface again, the process of closing the call recording is triggered.

[0337] S304b, the audio service class sends the close recording instruction to the audio hardware management module.

[0338] S304c, in response to receiving the close recording instruction, the audio hardware management module determines whether the first terminal is currently in virtual call.

[0339] In this embodiment, the process of determining whether the first terminal is currently in virtual call by the audio hardware management module in S304c is similar to that in S3043, which will not be repeated here.

[0340] S304d, if the first terminal is currently in virtual call, the audio hardware management module does not close the first PCM device, and stops obtaining the recording data from the virtual call buffer area.

[0341] If the user closes the call recording when the first terminal 101 is in virtual call, since the virtual call needs to use the first PCM device, the audio hardware management module does not close the first PCM device, and stops reading the recording data from the virtual call buffer area, i.e. Figure 19 The data flow diagram shown in Figure 21 is switched to the data flow diagram shown in

[0342] S304e, if the first terminal is not currently in virtual call, the audio hardware management module closes the first PCM device, and stops obtaining the recording data from the first PCM device.

[0343] If the user closes call recording while the first terminal 101 is in the middle of a call, and since the call does not require the use of the first PCM device, the audio hardware management module closes the first PCM device and stops reading recording data from it. Figure 22 The data flow diagram shown has been switched to Figure 22 The data flow diagram shown.

[0344] To better explain the processes S304a to S304e above, please continue to refer to... Figure 8 The process of the first terminal 101 turning off recording may include:

[0345] S41, When the user performs the operation to turn off recording, for example, when the user clicks the "Record" icon in the call interface again, the call application responds to the operation by generating a command to turn off recording and sends the command to the audio service class;

[0346] S42, the audio service class will disable the recording command sent to the audio hardware management module;

[0347] S43, the audio hardware management module detects the virtual call flag bit;

[0348] S44, If the audio hardware management module detects that the virtual call flag bit is true, then the audio hardware management module will set the recording flag bit to false;

[0349] S45, if the audio hardware management module detects that the virtual call flag is false, the audio hardware management module shuts down the first PCM device and then sets the recording flag to false.

[0350] The above Figure 15 The content described involves scenarios where the first terminal 101 may be in a virtual call when it starts recording the call, and scenarios where the first terminal 101 may still be in a virtual call when it stops recording the call.

[0351] Those skilled in the art should understand that in some scenarios, the first terminal 101 may already be recording a call when it initiates a virtual call; or, the first terminal 101 may still be recording a call when it closes a virtual call. For example, in conjunction with... Figure 19 The user is talking to another person on their phone, and the phone is recording the call. The user clicks the "Connect" icon displayed on the phone to initiate multi-screen collaboration, and the phone starts a virtual call. For example, combined with... Figure 11 The phone and tablet are in multi-screen collaboration, and the user is talking to the other party on the phone while the phone is recording. The user clicks the "disconnect" icon displayed on the phone to close the virtual call.

[0352] Therefore, the process of starting or closing the virtual call by the first terminal 101 also needs to be considered.

[0353] The process of starting the virtual call by the first terminal 101, i.e., the process of the first terminal receiving the call downlink data sent by the call opposite terminal device and sending the call downlink data to the second terminal, the process of the second terminal playing the call downlink data in S302, S302-1, S302-2, S302-3 and S302-3, and the process of the second terminal collecting the first call uplink data and sending the first call uplink data to the call opposite terminal device through the first terminal in S303, can include S3021-S3025.

[0354] S3021, the call application program sends a start virtual call instruction to the cooperation module in response to the user's operation.

[0355] S3022, the cooperation module sends the start virtual call instruction to the virtual call hardware management module. S3023, in response to receiving the start virtual call instruction, the virtual call hardware management module determines whether the first terminal is currently performing call recording.

[0356] The virtual call hardware management module can determine whether the first terminal 101 is currently performing call recording through the recording mark bit introduced in S3042, i.e., the process of the virtual call hardware management module determining whether the first terminal is currently performing call recording in S3023 can include:

[0357] The virtual call hardware management module detects the recording mark bit; if the recording mark bit is in the first identification, the first terminal is currently performing call recording; if the recording mark bit is in the second identification, the first terminal is not currently performing call recording.

[0358] S3024, if the first terminal is currently performing call recording, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and disables the first PCM device to obtain uplink data, reads the call downlink data from the first PCM device and sends the call downlink data to the transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal and writes the first call uplink data into the second PCM device, and mixes the call downlink data and the first call uplink data to obtain recording data and stores the recording data into the virtual call buffer area.

[0359] If the user starts the virtual call when the first terminal 101 is currently performing call recording, i.e., as shown in Figure 21As shown, the audio processor receives downlink call data sent by the other end of the call and sends it to the speaker for playback. The audio processor also acquires the second uplink call data collected by the microphone and sends it to the other end of the call. The first PCM device acquires the downlink call data and the second uplink call data from the audio processor.

[0360] Because virtual calls require switching the call audio from the first terminal 101 to the second terminal 102, call recording requires the audio hardware management module to obtain the recording data, i.e., such as... Figure 20 As shown, the virtual call hardware management module needs to disable the local audio device and prevent the first PCM device from obtaining uplink data from the audio processor. Then, the virtual call hardware management module reads downlink call data from the first PCM device, writes the first uplink call data to the second PCM device, and mixes the downlink call data and the first uplink call data to obtain recording data and stores it in the virtual call buffer.

[0361] S3025, if the first terminal is not currently recording a call, the virtual call hardware management module disables the audio device of the first terminal, and opens the first PCM device and prevents the first PCM device from acquiring uplink data, and reads downlink call data from the first PCM device and sends it to the transmission module of the first terminal, and receives the first uplink call data sent by the transmission module of the first terminal and writes it to the second PCM device.

[0362] If a user initiates a virtual call while the first terminal 101 is already in a call, i.e., as... Figure 23 As shown, the audio processor receives downlink call data sent by the other end of the call and sends it to the speaker for playback; and the audio processor acquires the second uplink call data collected by the microphone and sends it to the other end of the call.

[0363] Because virtual calls require switching the call audio from the first terminal 101 to the second terminal 102, that is, as follows: Figure 23 As shown, the virtual call hardware management module needs to disable the local audio device and prevent the first PCM device from obtaining uplink data from the audio processor. Then, the virtual call hardware management module reads the downlink call data from the first PCM device and writes the first call uplink data to the second PCM device.

[0364] To better explain the processes S3021 to S3025 above, the following will be explained... Figure 6 to 10 The interactive diagram shown is used to describe the interaction, such as Figure 11 As shown, the process of the first terminal 101 initiating a virtual call may include:

[0365] S11, when the user performs the operation to start a virtual call, for example, the user performs...Figure 19 In response to the starting virtual call operation, the call application generates a starting virtual call instruction and sends the starting virtual call instruction to the cooperation module;

[0366] S12, the cooperation module sends the starting virtual call instruction to the virtual call hardware management module;

[0367] S1a, the virtual call hardware management module sets the virtual call flag to true;

[0368] S13, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and prohibits the first PCM device from obtaining uplink data; wherein the virtual call hardware management module can specify the first PCM device by usecase_incall_rec_uplink_and_downlink and prohibit the first PCM device from obtaining uplink data;

[0369] S15-1~S15-3, the audio processor receives the call downlink data sent by the call opposite terminal device, the first PCM device obtains the call downlink data from the audio processor, and the virtual call hardware management module reads the call downlink data from the first PCM device;

[0370] S16-1~S16-3, the audio device of the second terminal collects the first call uplink data and transmits the first call uplink data to the transmission module of the second terminal, the transmission module of the second terminal sends the first call uplink data to the transmission module of the first terminal, and the transmission module of the first terminal sends the first call uplink data to the virtual call hardware management module;

[0371] S1b, the virtual call hardware management module detects the recording flag;

[0372] S1c, if the virtual call hardware management module detects that the recording flag is true, the following steps are performed:

[0373] S17-1~S17-2, the virtual call hardware management module mixes the first call uplink data and the call downlink data, and stores the mixed data in the virtual call cache area; and S15-4~S15-6, the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, the transmission module of the second terminal sends the call downlink data to the audio device of the second terminal, and the audio device of the second terminal plays the call downlink data; and S16-4~S16-5, the virtual call hardware management module writes the first call uplink data into the second PCM device, the audio processor reads the first call uplink data from the second PCM device and sends it to the call opposite terminal device.

[0374] S1d, if the virtual call hardware management module detects that the recording flag bit is false, then the following is performed:

[0375] S15-4~S15-6, the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, the transmission module of the second terminal sends the call downlink data to the audio device of the second terminal, and the audio device of the second terminal plays the call downlink data; and S16-4~S16-5, the virtual call hardware management module writes the first call uplink data into the second PCM device, and the audio processor reads the first call uplink data from the second PCM device and sends it to the call opposite device.

[0376] Optionally, the virtual call hardware management module can write the call downlink data and the first call uplink data into two buffer areas respectively, and then sequentially read the ith call downlink data and the ith first call uplink data from the two buffer areas for addition, so as to complete the mixing.

[0377] Further, the process of closing the virtual call by the first terminal 101 is introduced. After the above S3021~S3025, the call method of the multi-screen cooperation provided by the embodiment of the application can further include S302a~S302e.

[0378] S302a, the call application program sends a closing virtual call instruction to the cooperation module in response to the user's operation.

[0379] S302b, the cooperation module sends the closing virtual call instruction to the virtual call hardware management module.

[0380] S302c, in response to receiving the closing virtual call instruction, the virtual call hardware management module determines whether the first terminal is currently performing call recording.

[0381] In the embodiment, the process of determining whether the first terminal is currently performing call recording by the virtual call hardware management module in S302c is similar to that in S3023 described above, and will not be described here.

[0382] S302d, if the first terminal is currently performing call recording, the virtual call hardware management module releases the disabling of the audio device of the first terminal and cancels the prohibition of the first PCM device from obtaining uplink data.

[0383] If the user closes the virtual call when the first terminal 101 is performing call recording, since the call recording needs to use the first PCM device and the first PCM device needs to obtain uplink data and downlink data, the virtual call hardware management module releases the disabling of the local audio device and cancels the prohibition of the first PCM device from obtaining uplink data from the audio processor, i.e., from the data flow diagram shown in Figure 20 to Figure 21The data flow diagram shown.

[0384] S302e, if the first terminal is not currently recording the call, the virtual call hardware management module releases the disabling of the audio device of the first terminal and closes the first PCM device.

[0385] If the user closes the virtual call while the first terminal 101 is in a call, since the call does not need to use the first PCM device, the virtual call hardware management module releases the disabling of the local audio device and closes the first PCM device, i.e., from Figure 23 The data flow diagram shown is switched to Figure 14 to Figure 17 The data flow diagram shown.

[0386] To better illustrate the processes of S302a-S302e, please continue to refer to Figure 24 The process in which the first terminal 101 closes the virtual call can include:

[0387] S31, when the user performs a closing virtual call operation, for example, the user performs any one of the operations in ​ , the call application generates a closing virtual call instruction in response to the operation and issues the closing virtual call instruction to the coordination module;

[0388] S32, the coordination module issues the closing virtual call instruction to the virtual call hardware management module;

[0389] S3a, the virtual call hardware management module detects the recording flag bit;

[0390] S3b, if the virtual call hardware management module detects that the recording flag bit is true, then:

[0391] S33, the virtual call hardware management module cancels the prohibition of the first PCM device from obtaining uplink data, and releases the disabling of the audio device of the first terminal; and S35-1-S35-2, the audio device of the first terminal collects second call uplink data and transmits the second call uplink data to the audio processor, the audio processor sends the second call uplink data to the call opposite terminal device; and S36-1-S36-2, the audio processor receives the call downlink data sent by the call opposite terminal device and transmits it to the audio device of the first terminal for playing; and S37-1-S37-2, the first PCM device obtains the call downlink data and the second call uplink data from the audio processor and performs mixing; and S3c, the virtual call hardware management module sets the virtual call flag bit to false.

[0392] S3d, if the virtual call hardware management module detects that the recording flag bit is false, then:

[0393] S3e, the virtual call hardware management module closes the first PCM device; and S33, the virtual call hardware management module un-disables the microphone and the speaker, S35-1~S35-2, the audio device of the first terminal collects the second call uplink data and transmits the second call uplink data to the audio processor, the audio processor sends the second call uplink data to the call opposite end device; and S36-1~S36-2, the audio processor receives the call downlink data sent by the call opposite end device and transmits the call downlink data to the audio device of the first terminal for playing; S3f, the virtual call hardware management module sets the virtual call flag position to false.

[0394] The above introduces the multi-screen cooperative call method of the call system 100 applied to multi-screen cooperation in the embodiment of the present application, and the multi-screen cooperative call method applied to the first terminal 101 and the second terminal 102 is introduced respectively as follows.

[0395] The embodiment of the present application further provides a multi-screen cooperative call method, which is applied to the first terminal 101, for example, a mobile phone, and can include the following steps.

[0396] S401, the first terminal receives the incoming call of the call opposite end device, wherein the first terminal is connected with the second terminal and the second terminal displays the first terminal interface;

[0397] S402, in response to the user's call answering operation on the first terminal interface in the second terminal, the call downlink data sent by the call opposite end device is received and sent to the second terminal, so that the second terminal plays the call downlink data;

[0398] S403, the first call uplink data collected by the second terminal is received and sent to the call opposite end device;

[0399] S404, in response to the user's opening recording operation, the recording data is obtained, wherein the recording data includes the call downlink data and the first call uplink data.

[0400] The detailed implementation of S401~S404 can be referred to the description of the foregoing embodiment, which will not be described here.

[0401] The embodiment of the present application further provides a multi-screen cooperative call method, which is applied to the second terminal 102, for example, a tablet computer, and can include the following steps.

[0402] S501, Receive and play downlink call data sent by the first terminal. The downlink call data is sent by the other end of the call to the first terminal after the first terminal receives an incoming call from the other end of the call and responds to the user's answering operation on the first terminal interface in the second terminal, when the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal.

[0403] S502, the second terminal collects the first call uplink data and sends the first call uplink data to the other end device of the call through the first terminal, so that the first terminal responds to the user's recording operation and obtains the recording data, wherein the recording data includes the call downlink data and the first call uplink data.

[0404] For detailed implementation of S501 to S502 above, please refer to the description of the previous embodiments, which will not be repeated here.

[0405] This application also provides a terminal, such as... ​ As shown, the terminal 600 may include one or more processors 601, a memory 602, an audio device 603, and a display screen 604, with the processor 601, memory 602, audio device 603, and display screen 604 coupled together. The terminal 600 includes a first terminal 101 or a second terminal 102. The memory 602 stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the terminal 600 can perform the various functions or steps executed by the first terminal 101 in the above method embodiments, or the various functions or steps executed by the second terminal 102 in the above method embodiments.

[0406] The memory 602 may include high-speed random access memory (RAM) or non-volatile memory (NVM).

[0407] Processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by the integrated logic circuitry in the hardware of processor 601 or by software instructions. The processor 601 described above can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), embedded ARM chips, etc.

[0408] The embodiment of the present application further provides a chip system, which comprises at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a line. For example, the interface circuit can be used to receive signals from other devices (such as the memory of the terminal). For another example, the interface circuit can be used to send signals to other devices (such as the processor). Illustratively, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the terminal can perform various steps in the above embodiment. Of course, the chip system can also include other discrete devices, and the embodiment of the present application does not make specific limitations thereto.

[0409] The embodiment of the present application further provides a computer storage medium, which comprises computer instructions. When the computer instructions are run on the terminal, the terminal can perform various functions or steps performed by the first terminal 101 in the above method embodiment, or various functions or steps performed by the second terminal 102 in the above method embodiment.

[0410] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer can perform various functions or steps performed by the first terminal 101 in the above method embodiment, or various functions or steps performed by the second terminal 102 in the above method embodiment.

[0411] The functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0412] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or partially, or all or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, and includes several instructions for causing an apparatus (which can be a single-chip machine, a chip, etc.) or a processor to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0413] The above merely illustrates the specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any change or replacement within the technical scope disclosed by the embodiments of the present application should be covered in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-screen collaborative call method, characterized in that, The method includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, the first terminal receives an incoming call from the other end of the call. In response to a user's operation of answering a call on the interface of the first terminal in the second terminal, the first terminal receives downlink call data sent by the other end of the call and sends it to the second terminal, and the second terminal plays the downlink call data. The second terminal collects the first call uplink data and sends the first call uplink data to the call peer device through the first terminal. The first terminal includes a first PCM device, a virtual call hardware management module, a second PCM device, and an audio processor. The virtual call hardware management module is used to prevent the first PCM device from acquiring uplink data and to acquire the call downlink data from the first PCM device and send it to the second terminal. In response to the user's initiation of recording, the first terminal obtains recording data, wherein the recording data includes the downlink data of the call and the uplink data of the first call. The first terminal also includes a virtual call buffer, which is used to store the downlink data of the call and the uplink data of the first call. The steps of the second terminal collecting the first call uplink data and sending the first call uplink data to the call peer device through the first terminal include: The second terminal collects the first call uplink data and sends the first call uplink data to the virtual call hardware management module; The virtual call hardware management module writes the first call uplink data into the second PCM device; The audio processor reads the first call uplink data from the second PCM device and sends it to the call peer device.

2. The method as described in claim 1, characterized in that, The method further includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, in response to the user's dialing operation on the interface of the first terminal in the second terminal, the first terminal calls the other end device of the call; After the other end of the call is connected, the first terminal receives the downlink call data sent by the other end of the call and sends it to the second terminal, and the second terminal plays the downlink call data. The second terminal collects the first call uplink data and sends the first call uplink data to the other end device through the first terminal.

3. The method as described in claim 1, characterized in that, The method further includes: When the first terminal is in a call with the other end device, the first terminal displays a pop-up window, which prompts the user to enable multi-screen collaboration between the first terminal and the second terminal. In response to the user's operation on the connection icon in the pop-up window, the first terminal connects with the second terminal and the second terminal displays the interface of the first terminal. The first terminal receives downlink call data sent by the other end device and sends it to the second terminal. The second terminal plays the downlink call data. The second terminal collects the first call uplink data and sends the first call uplink data to the other end device through the first terminal.

4. The method as described in claim 1, characterized in that, The method further includes: When the first terminal is in a call with the other end device, and when the first terminal is connected to the second terminal and the second terminal displays the first terminal interface, the first terminal displays the first notification panel. In response to the user's operation on the audio / video switching icon in the first notification panel on the first terminal, the first terminal receives downlink call data sent by the other end device and sends it to the second terminal, and the second terminal plays the downlink call data; The second terminal collects the first call uplink data and sends the first call uplink data to the other end device through the first terminal.

5. The method as described in claim 1, characterized in that, The method further includes: When the first terminal is in a call with the other end device, and when the first terminal is connected to the second terminal and the second terminal displays the first terminal interface, the second terminal displays the first notification panel. In response to the user's operation on the audio / video switching icon in the first notification panel on the second terminal, the first terminal receives downlink call data sent by the other end device and sends it to the second terminal, and the second terminal plays the downlink call data; The second terminal collects the first call uplink data and sends the first call uplink data to the other end device through the first terminal.

6. The method according to any one of claims 1-5, characterized in that, The step of the first terminal receiving downlink call data sent by the other end of the call and sending it to the second terminal, and the second terminal playing the downlink call data, includes: The audio processor receives the downlink data of the call sent by the peer device; The first PCM device obtains the downlink call data from the audio processor; The second terminal plays the downlink data of the call.

7. The method as described in claim 1, characterized in that, The first terminal also includes an audio hardware management module; The step of the first terminal receiving downlink call data sent by the peer device and sending it to the second terminal, and the second terminal playing the downlink call data, further includes: The virtual call hardware management module prevents the first PCM device from obtaining the first call uplink data from the audio processor; The step of the first terminal obtaining recording data in response to the user's recording initiation operation includes: In response to the user's initiation of recording, the audio hardware management module obtains the recording data from the virtual call buffer, wherein the recording data is stored by the virtual call hardware management module after mixing the downlink call data and the first uplink call data.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, and the user is making a call with the other end device through the second terminal, in response to the user's closing operation of the first terminal interface in the second terminal, the first terminal plays the downlink data of the call and collects the uplink data of the call and sends it to the other end device. The first terminal obtains recording data, wherein the recording data includes the downlink data of the call and the uplink data of the second call.

9. The method as described in claim 8, characterized in that, The method further includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, and the user is making a call with the other end device through the second terminal, the first terminal displays the second notification panel. In response to the user's operation of disconnecting the icon in the second notification panel on the first terminal, the first terminal plays the downlink call data and collects the uplink call data to send to the other end device.

10. The method as described in claim 8, characterized in that, The method further includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, and the user is making a call with the other end device through the second terminal, the first terminal displays the second notification panel. In response to the user's operation on the audio / video switching icon in the second notification panel on the first terminal, the first terminal plays the downlink call data and collects the uplink call data and sends it to the other end device.

11. The method as described in claim 8, characterized in that, The method further includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, and the user is making a call with the other end device through the second terminal, the second terminal displays a second notification panel. In response to the user's operation on the audio / video switching icon in the second notification panel on the second terminal, the first terminal plays the downlink call data and collects the uplink call data to send to the other end device.

12. The method as described in claim 8, characterized in that, The method further includes: When the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal, and the user is making a call with the other end device through the second terminal, if the first terminal detects that it is close to the user's ear, the first terminal plays the downlink data of the call and collects the uplink data of the call and sends it to the other end device.

13. The method according to any one of claims 9-12, characterized in that, The first terminal includes: an audio device; The steps of the first terminal playing the downlink data of the call and collecting the second uplink data of the call and sending it to the other end device include: The audio processor receives the downlink call data and plays it through the audio device of the first terminal, and acquires the second uplink call data collected by the audio device of the first terminal and sends it to the other end device of the call.

14. The method as described in claim 13, characterized in that, The first terminal also includes an audio hardware management module; The step of the first terminal playing the downlink data of the call and collecting the second uplink data of the call and sending it to the other end device of the call further includes: The virtual call hardware management module cancels the restriction on the first PCM device from obtaining uplink data; The steps for the first terminal to obtain the recording data include: The audio hardware management module reads the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device from the audio processor by obtaining the downlink call data and the uplink call data and mixing them.

15. The method according to any one of claims 9-12, characterized in that, The first terminal also includes: an audio hardware management module and an audio processor; The step of the first terminal obtaining recording data in response to the user's recording initiation operation includes: In response to the user's activation of recording, the audio hardware management module determines whether the first terminal is currently conducting a virtual call; If the first terminal is currently making a virtual call, the audio hardware management module obtains the recording data from the virtual call buffer, wherein the recording data is stored by the virtual call hardware management module after mixing the downlink data of the call and the uplink data of the first call; If the second terminal is not currently making a virtual call, the audio hardware management module opens the first PCM device and reads the recording data from the first PCM device. The recording data is obtained by the first PCM device from the audio processor by obtaining the downlink data of the call and the uplink data of the second call and mixing them.

16. The method as described in claim 15, characterized in that, The method further includes: In response to the user's action to turn off recording, the audio hardware management module determines whether the first terminal is currently conducting a virtual call; If the first terminal is currently making a virtual call, the audio hardware management module does not shut down the first PCM device and stops retrieving the recording data from the virtual call buffer. If the first terminal is not currently making a virtual call, the audio hardware management module shuts down the first PCM device and stops acquiring the recording data from the first PCM device.

17. The method as described in claim 16, characterized in that, The virtual call hardware management module maintains a virtual call flag bit, which has a first state and a second state. The first state indicates that the first terminal is currently making a virtual call, and the second state indicates that the first terminal is not currently making a virtual call. The step of the audio hardware management module determining whether the first terminal is currently making a virtual call includes: The audio hardware management module detects the virtual call flag bit; If the virtual call flag is in the first state, then the first terminal is currently making a virtual call; If the virtual call flag is in the second state, then the first terminal is not currently making a virtual call.

18. The method as described in claim 17, characterized in that, The audio hardware management module maintains recording marker bits; The method further includes: In response to the user's initiation of recording, the audio hardware management module sets the recording marker position as a first identifier, the first identifier indicating that the first terminal is currently recording a call; In response to the user's action to turn off recording, the audio hardware management module sets the recording location to a second identifier, wherein the second identifier indicates that the first terminal is not currently recording a call.

19. The method according to any one of claims 9-12, characterized in that, The first terminal also includes: a second PCM device and an audio device; The steps of the first terminal receiving downlink call data sent by the peer device and sending it to the second terminal, the second terminal playing the downlink call data, and the second terminal collecting first uplink call data and sending the first uplink call data to the peer device through the first terminal include: In response to the user's virtual call activation operation, the virtual call hardware management module determines whether the first terminal is currently recording a call. If the first terminal is currently recording a call, the virtual call hardware management module disables the audio device of the first terminal, and opens the first PCM device and prevents the first PCM device from acquiring uplink data, and reads the downlink data of the call from the first PCM device and sends it to the second terminal, and receives the first uplink data of the call sent by the second terminal and writes it to the second PCM device, and mixes the downlink data of the call and the first uplink data of the call to obtain the recording data and stores it in the virtual call buffer. If the first terminal is not currently recording a call, the virtual call hardware management module disables the audio device of the first terminal, and opens the first PCM device and prevents the first PCM device from acquiring uplink data, and reads the downlink call data from the first PCM device and sends it to the second terminal, and receives the first uplink call data sent by the second terminal and writes it to the second PCM device.

20. The method as described in claim 19, characterized in that, The steps of the first terminal playing the downlink data of the call and collecting the second uplink data of the call and sending it to the other end device include: In response to the user's action to close the virtual call, the virtual call hardware management module determines whether the first terminal is currently recording a call. If the first terminal is currently recording a call, the virtual call hardware management module removes the audio device from the first terminal and cancels the restriction on the first PCM device to obtain uplink data. If the first terminal is not currently recording a call, the virtual call hardware management module unlocks the audio device of the first terminal and shuts down the first PCM device.

21. The method as described in claim 19, characterized in that, The first terminal also includes an audio hardware management module, which maintains a recording flag bit. The recording flag bit has a first identifier and a second identifier. The first identifier indicates that the first terminal is currently recording a call, and the second identifier indicates that the first terminal is not currently recording a call. The step of the virtual call hardware management module determining whether the first terminal is currently recording a call includes: The virtual call hardware management module detects the recording flag bit; If the recording flag is in the first identifier, then the first terminal is currently recording the call; If the recording flag is in the second flag, then the first terminal is not currently recording a call.

22. The method as described in claim 21, characterized in that, The virtual call hardware management module maintains a virtual call flag bit, and the method further includes: In response to the user's virtual call activation operation, the virtual call hardware management module marks the virtual call as a first state, the first state indicating that the first terminal is currently conducting a virtual call; In response to the user's action to close the virtual call, the virtual call hardware management module marks the virtual call as a second state, which indicates that the first terminal is not currently conducting a virtual call.

23. A multi-screen collaborative call method, characterized in that, Applied to a first terminal, the method includes: The first terminal receives an incoming call from the other end of the call, wherein the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal; In response to a user's operation of answering a call on the interface of the first terminal in the second terminal, the system receives downlink call data sent by the other end of the call and sends it to the second terminal so that the second terminal can play the downlink call data. The system receives first uplink call data collected by the second terminal and sends the first uplink call data to the peer device. The first terminal includes a first PCM device, a virtual call hardware management module, a second PCM device, and an audio processor. The virtual call hardware management module is used to prevent the first PCM device from acquiring uplink data and to acquire downlink call data from the first PCM device and send it to the second terminal. In response to the user's recording operation, the recording data is obtained, wherein the recording data includes the downlink data of the call and the first uplink data of the call, and the first terminal further includes a virtual call buffer, which is used to store the downlink data of the call and the first uplink data of the call; The steps of the second terminal collecting the first call uplink data and sending the first call uplink data to the call peer device through the first terminal include: The second terminal collects the first call uplink data and sends the first call uplink data to the virtual call hardware management module; The virtual call hardware management module writes the first call uplink data into the second PCM device; The audio processor reads the first call uplink data from the second PCM device and sends it to the call peer device.

24. A multi-screen collaborative call method, characterized in that, Applied to a second terminal, the method includes: The system receives and plays downlink call data sent by a first terminal. The downlink call data is sent by the peer device to the first terminal after the first terminal receives an incoming call from the peer device and responds to the user's answering operation on the first terminal interface in the second terminal, when the first terminal is connected to the second terminal and the second terminal displays the interface of the first terminal. The first terminal includes a first PCM device, a virtual call hardware management module, a second PCM device, and an audio processor. The virtual call hardware management module is used to prevent the first PCM device from acquiring uplink data and to acquire the downlink call data from the first PCM device and send it to the second terminal. The second terminal collects the first call uplink data and sends the first call uplink data to the virtual call hardware management module; the virtual call hardware management module writes the first call uplink data into the second PCM device; The audio processor reads the first call uplink data from the second PCM device and sends it to the call peer device, so that the first terminal responds to the user's recording start operation and obtains recording data. The recording data includes the call downlink data and the first call uplink data. The first terminal also includes a virtual call buffer, which is used to store the call downlink data and the first call uplink data.

25. A terminal, characterized in that, The terminal includes: One or more processors; Memory; Audio equipment; Display screen; The memory is used to store computer program code, which includes computer instructions; when the processor executes the computer instructions, the terminal executes the multi-screen collaborative call method executed by the first terminal as described in any one of claims 1-22, or the terminal executes the multi-screen collaborative call method executed by the second terminal as described in any one of claims 1-22.

26. A multi-screen collaborative calling system, characterized in that, The system includes a first terminal and a second terminal, wherein the first terminal executes the multi-screen collaborative call method as described in any one of claims 1-22, and the second terminal executes the multi-screen collaborative call method as described in any one of claims 1-22.

27. A computer-readable storage medium, characterized in that, The method includes a computer program that, when running on a terminal, causes the terminal to execute the multi-screen collaborative call method executed by the first terminal as described in any one of claims 1-22, or the terminal to execute the multi-screen collaborative call method executed by the second terminal as described in any one of claims 1-22.

Images