Audio stream processing method and device

Abstract

The embodiment of the invention provides an audio stream processing method and device, relates to the field of terminals, and can avoid the problem that user experience is affected due to conflicts between collaborative services and other services. The method is applied to a first electronic device, the first electronic device comprises an SIM card or an eSIM card, the first electronic device further comprises a microphone, and the method comprises the steps that the first electronic device collects a first audio stream based on the microphone and sends the first audio stream to a second electronic device, and the first electronic device and the second electronic device are in a multi-screen cooperation scene; the first electronic device monitors a call state of an SIM card or an eSIM card of the first electronic device; when the call state of the SIM card or the eSIM card of the first electronic equipment is an answering state, stopping collecting the first audio stream; and the first electronic equipment collects a second audio stream based on the microphone, and sends the second audio stream to the third electronic equipment based on the SIM card or the eSIM card.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to an audio stream processing method and apparatus. Background Technology

[0002] With the continuous development of terminal devices, collaboration between terminal devices is becoming increasingly common. Terminal device collaboration refers to the ability of terminal devices to share data in real time and collaboratively process the same business. For example, terminal devices can conduct collaborative calls.

[0003] However, when terminal devices collaborate, for example, when one terminal device (the controlled device) processes collaborative services for another terminal device (the controlling device), the controlled device itself may generate some services that conflict with the collaborative services, affecting the user experience. Summary of the Invention

[0004] This application provides an audio stream processing method and apparatus that can avoid the problem of user experience being affected by conflicts between collaborative services and other services.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an audio stream processing method is provided, applied to a first electronic device, the first electronic device including a SIM card or eSIM card, and the first electronic device also including a microphone, the method comprising: the first electronic device acquiring a first audio stream based on the microphone, and sending the first audio stream to a second electronic device, the first electronic device and the second electronic device being in a multi-screen collaboration scenario; the first electronic device monitoring the call status of the SIM card or eSIM card of the first electronic device; when the call status of the SIM card or eSIM card of the first electronic device is in an answering state, stopping the acquisition of the first audio stream; the first electronic device acquiring a second audio stream based on the microphone, and sending the second audio stream to a third electronic device based on the SIM card or eSIM card.

[0007] Based on the method provided in this application embodiment, during multi-screen collaboration between the first electronic device and the second electronic device, the first electronic device can monitor the call status of its own SIM card or eSIM card. When the call status of the first electronic device's SIM card or eSIM card is in an answering state, the first electronic device can stop collecting the first audio stream, that is, stop collecting the audio stream for the multi-screen collaboration service, and can instead collect the audio stream (second audio stream) for the operator's call service based on the microphone, thereby ensuring normal communication between the first electronic device and the remote electronic device (third electronic device). This avoids the problem of the collaboration service and the operator's call service simultaneously using the microphone, which could lead to mutual interference and a degraded user experience.

[0008] In one possible implementation, the multi-screen collaboration scenario includes a collaborative call scenario, where collaborative calls include VoIP calls and / or SIM card calls. That is, during a collaborative call (VoIP or SIM card call) between a first electronic device and a second electronic device, the first electronic device can monitor the call status of its own SIM card or eSIM card. When the call status of the first electronic device's SIM card or eSIM card is in an answered state, the first electronic device can stop collecting the first audio stream, i.e., stop collecting audio streams for the collaborative call service, and instead collect audio streams (second audio streams) for the carrier call service using its microphone, thus ensuring normal communication between the first electronic device and the remote electronic device (third electronic device). This avoids the problem of collaborative call services and carrier call services simultaneously using the microphone, which could cause mutual interference and thus degrade the user experience.

[0009] In one possible implementation, the multi-screen collaboration scenario includes collaborative audio recording and / or collaborative speech recognition. When the multi-screen collaboration scenario is a collaborative audio recording scenario, the method provided in this application embodiment can not only prevent the first electronic device (e.g., mobile phone B) from sending the audio data of the operator's call content to the second electronic device (e.g., mobile phone A), thus preventing the leakage of the operator's call content of the first electronic device and ensuring the security of the operator's call, but also prevent the first electronic device (mobile phone B) from sending the audio data of the operator's call content to the second electronic device (mobile phone A), thus preventing the operator's call content of the first electronic device from interfering with the recording content of the second electronic device. Therefore, the method provided in this application embodiment can simultaneously guarantee the user experience of both the second electronic device and the first electronic device.

[0010] When the multi-screen collaboration scenario involves collaborative voice recognition, the method provided in this application not only prevents the leakage of the carrier call content of the first electronic device, ensuring the security of the carrier call, but also prevents the first electronic device (phone B) from sending the audio data of the carrier call content to the second electronic device (phone A), thus avoiding interference between the carrier call content of the first electronic device and the voice recognition content of the second electronic device. For example, in the case of a smart voice assistant scenario, the first electronic device (phone B) not sending the audio data of the carrier call content to the second electronic device (phone A) can prevent the second or first electronic device from misinterpreting the carrier call content as audio commands to control the smart voice assistant, leading to misoperation.

[0011] In one possible implementation, after stopping the acquisition of the first audio stream, the method further includes: the first electronic device displaying a first pop-up window, which is used to notify the user that the audio stream in the multi-screen collaboration scenario has been disconnected. In this way, the user is notified of the disconnection of the audio stream in the multi-screen collaboration scenario via the first pop-up window, thus avoiding impacting the user experience.

[0012] In one possible implementation, before the first electronic device acquires the first audio stream based on the microphone, the method further includes: responding to a first user operation, starting a Distributed Device Virtualization (DMSDP) process; the DMSDP process initializes a virtual audio proxy module, which includes an audio state adjustment module; the audio state adjustment module initializes the microphone's state to a first state, which indicates that the microphone is available; the first operation triggers multi-screen collaboration; when the microphone's state is in the first state, the virtual audio proxy module acquires the first audio stream based on the microphone; the DMSDP process creates a call state monitoring module, which monitors the call state of the first electronic device's SIM card or eSIM card; when the call state monitoring module detects that the first electronic device's SIM card or eSIM card's call state is in an answering state, it notifies the audio state adjustment module, which marks the microphone's state to a second state, which indicates that the microphone is unavailable; when the microphone's state is in the second state, the virtual audio proxy module stops acquiring the first audio stream.

[0013] Based on the method provided in this application embodiment, during multi-screen collaboration between the first electronic device and the second electronic device, the call status monitoring module of the first electronic device can monitor the call status of its own SIM card or eSIM card. When the call status monitoring module detects that the SIM card or eSIM card is in an answered call state, it can promptly notify the audio status management module of the call status of the SIM card or eSIM card. The audio status management module can refresh the microphone status based on the call status of the SIM card or eSIM card (e.g., refresh to an unavailable state). When the virtual audio proxy module determines that the microphone status is unavailable, it can stop collecting the first audio stream, i.e., stop collecting the audio stream for the multi-screen collaboration service. This allows the operator's call service to collect the audio stream (second audio stream) based on the microphone, ensuring normal communication between the first electronic device and the remote electronic device (third electronic device). This avoids the problem of multi-screen collaboration and operator's call services using the microphone simultaneously, which could cause mutual interference and a degraded user experience.

[0014] In one possible implementation, SIM card or eSIM-based call services have higher priority than multi-screen collaboration services. This means that microphone access can be preempted based on service priority; a higher-priority service (or application) can preempt a lower-priority service (or application) from using the microphone.

[0015] In one possible implementation, the method further includes: if the call status monitoring module detects that the call status of the SIM card or eSIM card of the first electronic device is in an "off" state, it notifies the audio status adjustment module, which then re-marks the microphone status as the first state; while the microphone status is in the first state, the virtual audio proxy module continues to collect the first audio stream based on the microphone. Thus, even after the call service ends, the virtual audio proxy module can continue to collect the first audio stream for the multi-screen collaboration service based on the microphone, allowing the multi-screen collaboration service to continue normally.

[0016] In one possible implementation, the call status monitoring module is destroyed after the multi-screen collaboration scenario ends. It's understandable that once the multi-screen collaboration scenario ends, i.e., the multi-screen collaboration service is shut down, the conflict scenario no longer exists (the collaboration service and the carrier call service cannot coexist, therefore no conflict occurs). Destroying the call monitoring thread at this point not only prevents the collaboration service and the carrier call service from interfering with each other, but also clears memory and saves power.

[0017] In one possible implementation, the first electronic device is a mobile phone or a tablet computer.

[0018] Secondly, an audio stream processing method is provided, applied to a system including a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are in a multi-screen collaborative state, the first electronic device includes a SIM card or an eSIM card, and the first electronic device also includes a microphone, the method comprising: the first electronic device acquiring a first audio stream based on the microphone and sending the first audio stream to the second electronic device; the second electronic device receiving the first audio stream; the first electronic device monitoring the call status of the SIM card or eSIM card of the first electronic device; stopping the acquisition of the first audio stream when the call status of the SIM card or eSIM card of the first electronic device is in an answering state; and the first electronic device acquiring a second audio stream based on the microphone and sending the second audio stream to a third electronic device based on the SIM card or eSIM card.

[0019] Based on the method provided in this application embodiment, during multi-screen collaboration between the first electronic device and the second electronic device, the first electronic device can monitor the call status of its own SIM card or eSIM card. When the call status of the first electronic device's SIM card or eSIM card is in an answering state, the first electronic device can stop collecting the first audio stream, that is, stop collecting the audio stream for the multi-screen collaboration service, and can instead collect the audio stream (second audio stream) for the operator's call service based on the microphone, thereby ensuring normal communication between the first electronic device and the remote electronic device (third electronic device). This avoids the problem of the collaboration service and the operator's call service simultaneously using the microphone, which could lead to mutual interference and a degraded user experience.

[0020] Thirdly, an audio stream processing system is provided, comprising a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are in a multi-screen collaborative state, the first electronic device includes a SIM card or an eSIM card, and the first electronic device also includes a microphone, wherein: the first electronic device is used to: acquire a first audio stream based on the microphone and send the first audio stream to the second electronic device; the second electronic device is used to: receive the first audio stream; the first electronic device is also used to: monitor the call status of the SIM card or eSIM card of the first electronic device; the first electronic device is also used to: stop acquiring the first audio stream when the call status of the SIM card or eSIM card is in the answering state; the first electronic device is also used to: acquire a second audio stream based on the microphone and send the second audio stream to a third electronic device based on the SIM card or eSIM card.

[0021] Based on the method provided in this application embodiment, during multi-screen collaboration between the first electronic device and the second electronic device, the first electronic device can monitor the call status of its own SIM card or eSIM card. When the call status of the first electronic device's SIM card or eSIM card is in an answering state, the first electronic device can stop collecting the first audio stream, that is, stop collecting the audio stream for the multi-screen collaboration service, and can instead collect the audio stream (second audio stream) for the operator's call service based on the microphone, thereby ensuring normal communication between the first electronic device and the remote electronic device (third electronic device). This avoids the problem of the collaboration service and the operator's call service simultaneously using the microphone, which could lead to mutual interference and a degraded user experience.

[0022] Fourthly, this application provides a chip system including one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The aforementioned chip system can be applied to electronic devices including communication modules and memory. The interface circuits are used to receive signals from the memory of the electronic device and send the received signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device (first electronic device) can perform the methods described in the first-third aspects and any possible design embodiments thereof.

[0023] Fifthly, this application provides a computer-readable storage medium comprising computer instructions. When executed on an electronic device (a first electronic device), the computer instructions cause the electronic device to perform the methods described in the first to third aspects and any possible design embodiments thereof.

[0024] Sixthly, embodiments of this application provide an audio stream processing apparatus, including a processor and a memory coupled together. The memory stores program instructions, which, when executed by the processor, cause the apparatus to implement the methods described in the first to third aspects and any possible design embodiments thereof. The apparatus may be an electronic device (a first electronic device); or it may be a component of an electronic device, such as a chip.

[0025] In a seventh aspect, embodiments of this application provide an audio stream processing apparatus, which can be divided into different logical units or modules according to their functions, each unit or module performing different functions, so that the apparatus performs the methods described in the first aspect to the third aspect and any possible design method thereof.

[0026] Eighthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the methods described in the first to third aspects and any possible design of the present application.

[0027] It is understood that the beneficial effects achieved by the chip system described in the fourth aspect, the computer-readable storage medium described in the fifth aspect, the apparatus described in the sixth and seventh aspects, and the computer program product described in the eighth aspect can be referred to the beneficial effects in the first to third aspects and any possible design embodiments thereof, which will not be repeated here. Attached Figure Description

[0028] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;

[0029] Figure 2A schematic diagram of the structure of a first electronic device or a second electronic device provided in the embodiments of this application;

[0030] Figure 3 A schematic diagram of the software architecture of a first electronic device and a second electronic device provided in an embodiment of this application;

[0031] Figure 4A A schematic diagram provided for an embodiment of this application;

[0032] Figure 4B This is yet another display schematic diagram provided for an embodiment of this application;

[0033] Figure 5 This is yet another display schematic diagram provided for an embodiment of this application;

[0034] Figure 6 This is yet another display schematic diagram provided for an embodiment of this application;

[0035] Figure 7 This is yet another display schematic diagram provided for an embodiment of this application;

[0036] Figure 8 A schematic diagram illustrating an audio stream processing method provided in an embodiment of this application;

[0037] Figure 9 A schematic diagram illustrating yet another audio stream processing method provided in an embodiment of this application;

[0038] Figure 10 This is yet another display schematic diagram provided for an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0040] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant concepts or technologies is given first:

[0041] Trust ring: Devices that are on the same local area network and logged into the same account or have "Smart Interconnection" enabled can join the same trust ring. Multiple electronic devices in the trust ring are interconnected and can work together to transfer content and share hardware and software resources.

[0042] Multi-screen collaboration: Electronic devices within a trust loop can collaborate on multiple screens. Multi-screen collaboration refers to the coordinated processing of the same service by multiple electronic devices, such as collaborative calls. During a collaborative call, one electronic device (the master device) can collect audio data (sound pickup) using the microphone of another electronic device (the controlled device). The controlled device can then send the audio data collected by its microphone to the master device, which in turn sends it to the remote device in the call. In other words, the master device can use the microphone and other hardware resources of the controlled device to process collaborative services (e.g., collaborative calls).

[0043] Among them, the master control device can also be called the projection device or the calling device, and the controlled device can also be called the projected device or the calling device.

[0044] However, when multiple devices collaborate across screens, the controlled device itself may generate some services that conflict with the collaborative services, affecting the user experience.

[0045] In previous technologies, the controlling device could be a mobile phone, while the controlled device was typically a tablet, laptop, television, speaker, or similar device. In the latest technologies, the controlled device can also be a mobile phone. This scenario, where a mobile phone acts as the controlled device and collaborates with other devices on multiple screens, can be called a "one-device-multiple-uses" scenario. It's important to note that mobile phones themselves have carrier calling capabilities, meaning they can make calls using a SIM card or eSIM. When a mobile phone acts as the controlled device and collaborates with other devices (e.g., making collaborative calls), the collaborative services (e.g., collaborative calls) performed by the phone may conflict with its own carrier calling functionality.

[0046] Currently, in multi-screen collaboration scenarios, there are no restrictions on the use of the microphone on the projected device. When a mobile phone is simultaneously performing collaborative services (e.g., collaborative calls) and carrier calls, the microphone can be used in parallel. That is, collaborative services and carrier calls can use the microphone simultaneously, causing them to interfere with each other and resulting in a degraded user experience.

[0047] This application provides an audio stream processing method that can avoid the impact on user experience caused by conflicts between collaborative services and other services. For example, it can solve the problem of user experience being affected when audio devices (e.g., microphones) are used concurrently.

[0048] like Figure 1 As shown, the system architecture involved in this application embodiment may include a first electronic device (e.g., mobile phone B) and a second electronic device (e.g., mobile phone A). Mobile phone A and mobile phone B can form a network, and after networking, they can exchange information. The networking method can be a Wi-Fi (wireless fidelity) networking method, for example, as... Figure 1 As shown, mobile phone A and mobile phone B can connect to the same routing device (router), that is, connect to the same local area network. Alternatively, the networking method can be Bluetooth networking. Alternatively, Wi-Fi networking and Bluetooth networking methods can be used simultaneously, which is not limited in this application. Optionally, the system architecture involved in the embodiments of this application may also include more electronic devices, such as mobile phone C, tablet, etc., which is not limited in this application.

[0049] In this embodiment, after the first electronic device and the second electronic device form a network, they can authenticate each other using accounts (e.g., Honor accounts). Once the first and second electronic devices have completed networking and authentication, they can be considered to have joined a trust ring. Data access between electronic devices within the trust ring is more convenient and faster, and the security of data access between electronic devices can be guaranteed, preventing unauthorized data access.

[0050] In this embodiment of the application, the first electronic device can be a controlled device, and the second electronic device can be a master control device.

[0051] Taking mobile phone A as the main control device and mobile phone B as the controlled device as an example, after mobile phone A and mobile phone B establish a collaborative connection, mobile phone B can display the interface of mobile phone A. The user can operate on the interface of mobile phone A displayed on mobile phone A or mobile phone B to make mobile phone B perform corresponding functions (for example, make mobile phone B collect audio stream based on microphone and send it to mobile phone A).

[0052] The first electronic device can be a device with carrier calling functionality. For example, the first electronic device can be a mobile phone, tablet computer, wearable device (e.g., smartwatch), in-vehicle device, etc., with a SIM card or eSIM card installed. The embodiments of this application do not impose special limitations on the specific form of the first electronic device.

[0053] The second electronic device can be a mobile phone, personal computer (PC), tablet computer, desktop computer, handheld computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), television, audio equipment, camera, air conditioner, refrigerator, smart curtains, table lamp, chandelier, rice cooker, security equipment (such as smart electronic lock), robot, robot vacuum cleaner, smart scale, smart headphones, smart glasses, smart watch, smart bracelet, augmented reality (AR) / virtual reality (VR) device, in-vehicle equipment, etc. The embodiments of this application do not impose special restrictions on the specific form of the second electronic device.

[0054] The following explanation uses the hardware structure of electronic device 100 as an example, with the first electronic device or the second electronic device as an example. Figure 2 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0055] See Figure 2 Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management process 140, power management process 141, battery 142, antenna 1, antenna 2, mobile communication process 150, wireless communication process 160, audio process 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor process 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor process 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0056] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0057] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0058] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0059] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0060] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0061] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication process 150, wireless communication process 160, modem processor, and baseband processor.

[0062] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0063] The mobile communication process 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication process 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication process 150 can receive electromagnetic waves through antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication process 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0064] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194.

[0065] Wireless communication process 160 can provide solutions for wireless communication applications on electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. Wireless communication process 160 can be one or more devices integrating at least one communication processing process. Wireless communication process 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 110. Wireless communication process 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0066] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication process 150, and antenna 2 is coupled to wireless communication process 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology.

[0067] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0068] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as image playback), etc. The data storage area may store data created by electronic device 100 during use (such as audio data), etc.

[0069] The methods described in the following embodiments can all be implemented in the electronic device 100 having the above-described hardware structure.

[0070] The software system of the aforementioned electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 100.

[0071] The software architecture of the first electronic device and the second electronic device is described below. The software architecture of the first electronic device and the second electronic device may be different.

[0072] Figure 3 This application provides a software architecture for a first electronic device (the called device / controlled device) and a second electronic device (the calling device / master device). The first electronic device may include an application layer, an application framework layer, a hardware abstraction layer (HAL), and a kernel layer. The first electronic device may also include a system layer; this application does not limit the scope of the application.

[0073] The application layer of the first electronic device can support applications such as multi-screen collaboration and heterogeneous projection. Multi-screen collaboration applications enable multi-screen collaboration, allowing the controlled device to operate all content (the entire interface) of the controlling device. Heterogeneous projection applications allow partial content from the controlling device (e.g., the interface of the first application) to be projected onto the controlled device, allowing the controlled device to operate on that portion of the controlling device's content, while the controlling device can perform other tasks (e.g., displaying the interface of a second application).

[0074] The application layer of the first electronic device may also include a distributed mobile sensing development platform (DMSDP) interface. Applications in the application layer can interact with the DMSDP process in the application framework layer through the DMSDP interface. DMSDP can also be referred to as a distributed device virtualization module or a virtual modem.

[0075] Optionally, the application layer of the first electronic device may also include, but is not limited to, applications such as instant messaging, camera sharing, Bluetooth, gallery, calling, and maps.

[0076] The application framework layer of the first electronic device provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0077] The application framework layer of the first electronic device may include the DMSDP process, audio framework, etc.

[0078] The DMSDP process can include virtualization capabilities, a virtual audio proxy module, and a call status monitoring module.

[0079] The virtualization capability is the main thread of the DMSDP process, while the virtual audio proxy module and the call status monitoring module are two independent threads of the DMSDP process.

[0080] The virtual audio proxy module (also known as the virtual audio proxy thread) may include an audio state management module. This module can be used to mark the microphone's state so that the virtual audio proxy module can determine whether the microphone can be used based on that state.

[0081] The call status monitoring module (also known as the call status monitoring thread) can be used to monitor the call status of the SIM card or eSIM card of the first electronic device and promptly notify the audio status management module of the call status of the SIM card or eSIM card. The audio status management module can refresh the microphone status according to the call status of the SIM card or eSIM card.

[0082] The solution provided in this application embodiment does not involve much embedded modification. It adds an insertable call status monitoring thread to monitor the call status, supplemented by a notification mechanism after monitoring. The microphone status is marked by an audio status management module added to the virtual audio proxy module, thus achieving a method with minimal impact on the existing architecture.

[0083] It's important to note that placing the audio status management module within the virtual audio proxy module allows the virtual audio proxy module to promptly obtain the microphone's status and determine whether to use the microphone to capture the audio stream (first audio stream). However, if the audio status management module is placed as a separate thread outside the virtual audio proxy module, the communication between them (inter-thread communication) will incur significant latency. The virtual audio proxy module may not be able to promptly obtain the microphone's status, causing it to fail to stop capturing the audio stream in time. This can lead to interference between collaborative services and carrier call services. For example, if a carrier call has already started, and the virtual audio proxy module fails to stop capturing the audio stream in time, the multi-screen collaborative service may capture irrelevant call content, affecting the multi-screen collaborative service and reducing the security of the carrier call service. Therefore, placing the audio status management module within the virtual audio proxy module effectively reduces the latency for the virtual audio proxy module to obtain the microphone's status, allowing it to promptly determine whether to use the microphone to capture the audio stream (first audio stream) and avoid interference between collaborative services and carrier call services.

[0084] It should be noted that the personal information collected in the technical solution of this application (e.g., call status information) is limited to information with the individual consent of the user, including but not limited to notifying and reminding the user to read the relevant user agreement (notification) and sign the agreement (authorization) which includes the authorization of relevant user information before the user uses the function.

[0085] The audio framework supports various common audio encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The multimedia framework supports various common video formats and still image files. Supported video formats include, for example, AVI, WMV, FLV, and Blu-ray.

[0086] In some embodiments, the audio framework may also be located at the system layer, and this application does not specifically limit it.

[0087] The HAL layer is a wrapper around Linux kernel drivers, providing interfaces to the upper layers and shielding them from the implementation details of the lower-level hardware.

[0088] In this embodiment, the HAL layer of the first electronic device may include an audio HAL. The audio HAL is used to activate the local microphone according to the business requirements of the application layer, and to acquire audio streams (audio data) through the local microphone.

[0089] The kernel layer is the layer between hardware and software. In this embodiment, the kernel layer of the first electronic device may include an audio driver. The audio driver drives a microphone to activate it, allowing the electronic device to acquire audio streams (audio data) through the microphone head.

[0090] The kernel layer of the first electronic device may also include display drivers, camera drivers, sensor drivers, etc., which are not limited in this application.

[0091] The software architecture of the second electronic device may include an application layer, an application framework layer, a HAL layer, and a kernel layer. The second electronic device may also include a system layer; this application does not limit this.

[0092] The application framework layer of the second electronic device includes a DMSDP process, which incorporates virtualization capabilities. Data transmission for multi-screen collaborative services can be performed between the second and first electronic devices based on the virtualization services of the DMSDP process.

[0093] The descriptions of the application layer, HAL layer, and kernel layer of the second electronic device can be found in the descriptions of the application layer, HAL layer, and kernel layer of the first electronic device, and will not be repeated here.

[0094] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. In the description of this application, unless otherwise stated, "at least one" refers to one or more, and "more than one" refers to two or more. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0095] To facilitate understanding, the following example uses mobile phone A as the main control device and mobile phone B as the controlled device, with mobile phone A and mobile phone B working together, to introduce several possible connection methods for multi-screen collaboration.

[0096] 1. Establish connections through a trust ring.

[0097] like Figure 4A As shown in (a), in response to the user swiping down from the status bar of phone B to display the notification panel, the phone displays the notification panel, which includes the "Trust Ring Multi-Device Interconnection Center" option 41. In response to the user's triggering action (e.g., a click) on the "Trust Ring Multi-Device Interconnection Center" option 41, phone B can display as follows: Figure 4A The control interface 42 is shown in (b) above. This control interface 42 includes a device identifier 43 for the device (i.e., phone B) and a device identifier 44 for phone A (i.e., both phone B and phone A are joined to the same trust ring). The user can drag the identifier 44 of phone A towards the device identifier 43 of the device. When the drag operation ends near the device identifier 43 of the device, multi-screen collaboration is established between the device (i.e., phone B) and phone A.

[0098] 2. Establish a connection via Bluetooth.

[0099] In one embodiment, when a user wants to enable collaborative work between phone A and phone B, both Bluetooth and Wi-Fi can be turned on. Additionally, on phone A, the user can manually enable the multi-screen collaboration function. For example, the user can find the "Multi-device Collaboration" switch through "Settings" - "More Connections" - "Multi-device Collaboration" and turn it on, thus enabling the multi-screen collaboration function on phone A. Optionally, both phone A and phone B can be logged into the same account (e.g., a Honor account).

[0100] like Figure 4BAs shown in (a), the user swipes down from the status bar of phone B to access the notification panel, which includes a "Multi-screen Collaboration" option 31. The user can tap the "Multi-screen Collaboration" option 31. In response to the user's triggering of the "Multi-screen Collaboration" option 31, phone B displays a first prompt window, which includes first operation prompts instructing the user on how to operate. For example, as... Figure 4B As shown in (b), the first operation prompt includes the following: "1. Turn on your phone's Bluetooth and bring it close to this device. Once this device is detected, click 'Connect'. 2. After connecting, you can operate the phone on phone B to achieve data sharing between devices." Thus, the user can operate according to the first operation prompt in the first prompt window. For example, bringing phone A close to phone B.

[0101] In one example, as phone A approaches phone B, when phone A detects phone B, phone A displays a second notification window, such as... Figure 5 As shown in (a), the second prompt window includes a "Connect" option 41 and a "Cancel" option 42. In response to the user's triggering of the "Connect" option 41, phone A establishes a connection with phone B via Bluetooth. In another example, when phone A detects phone B while it is near phone B, the second prompt window may not be displayed; instead, phone A may automatically establish a connection with phone B via Bluetooth.

[0102] As an example, and not a limitation, during the process of phone A establishing a connection with phone B via Bluetooth, phone A can also display a third prompt window to indicate that the connection is in progress. For example, as... Figure 5 As shown in (b), the third prompt window can be window 43. Optionally, window 43 includes a "Cancel" option so that the user can cancel the connection during the connection establishment process if needed.

[0103] 3. Establish a connection by scanning a QR code.

[0104] On mobile device B, users can follow the path "My Phone" - "Connect Now" - "Scan to Connect" to operate on mobile device B. In response to the user's operation, mobile device B displays a QR code for establishing a connection. Optionally, mobile device B can also display a second operation prompt message to guide the user on how to operate, such as "Scan to connect using your mobile browser".

[0105] On mobile phone A, users can access an interface with a "Scan" option through a browser (or Smart Vision). In response to the user's activation of the "Scan" option, mobile phone A launches its camera, allowing the user to point the camera at the QR code displayed on mobile phone B to scan it.

[0106] In one example, after phone A successfully scans the QR code, it sends a connection request to phone B. Upon receiving the request, phone B can display a prompt message, such as "Device xx requests to establish a connection with this device. Do you agree to establish the connection?". Additionally, a fourth prompt window can include "Agree" and "Decline" options. In response to the user's "Agree" option, phone B establishes a connection with phone A.

[0107] It should be noted that the above explanation only illustrates the example of opening the QR code on mobile device B via the path "My Phone" - "Connect Now" - "Scan to Connect". In another embodiment, the QR code can also be opened via other paths. For an example, please refer to... Figure 4B In Figure (b), the first prompt window includes a first operation prompt and a second operation prompt, which reads "Can't find this device? You can also connect by scanning the QR code." The phrase "connect by scanning the QR code" is set to triggerable. Users can click on the "connect by scanning the QR code" content in the first prompt window. In response to the user's triggering of the "connect by scanning the QR code" action, the interface of phone B can display a QR code. Thus, the user can scan the QR code on phone B using phone A to establish a connection via scanning.

[0108] 4. Establish a connection by tapping.

[0109] Users can enable NFC and multi-screen collaboration on both phone A and phone B. Then, the user touches the NFC area on the back of phone A (around the rear camera) to the NFC area on the keyboard. In response to this action, phone A and phone B establish a connection via NFC. Optionally, before establishing the connection via NFC, both phone B and phone A can prompt the user to agree to the connection. After the user agrees, phone A and phone B will perform the connection establishment process. In one example, when phone A and phone B are connected, phone A can also notify the user via vibration or ringing.

[0110] It should be noted that the above-described possible connection methods are all illustrated using wireless methods as examples. In another embodiment, screen projection can also be performed via a wired method, such as through a Type-C to High Definition Multimedia Interface (HDMI) cable. This application does not limit the scope of this embodiment.

[0111] After phone A and phone B are successfully connected, as follows: Figure 6 As shown, phone B can mirror the window of phone A. This allows the user to perform operations within that window as needed.

[0112] In one example, when a user wants to make a video call via instant messaging software, they can click the icon of instant messaging software A in the window of phone B to open the instant messaging software, and then click the "Video Call" option within the instant messaging software. In response to the user's triggering of the "Video Call" option, phone B sends a video call control command to phone A. After receiving the video call control command, phone A initiates a video call request to conduct a video call with the other user. Please refer to [reference needed]. Figure 7 During this process, the screens of phones A and B are displayed synchronously.

[0113] In some embodiments, when mobile phone A connects (collaborates) to mobile phone B, it can use mobile phone B's camera to capture video footage and its microphone to capture audio data by default. That is, the controlling device can use the controlled device's camera, microphone, and other hardware resources to process collaborative services.

[0114] For ease of understanding, the audio stream processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0115] like Figure 8 As shown in the illustration, this application provides an audio stream processing method. The first electronic device is a mobile phone B, and the second electronic device is a mobile phone A. Mobile phone A is the source (master) and mobile phone B is the sink (controlled) as an example. The method includes:

[0116] 801. Mobile phone A and mobile phone B establish a connection and perform multi-screen collaboration.

[0117] After mobile phone A and mobile phone B establish a collaborative connection, mobile phone A and mobile phone B are in a multi-screen collaborative scenario and can perform multi-screen collaborative services.

[0118] Mobile phone A and mobile phone B can achieve multi-screen collaboration in a variety of ways, such as through a trust loop, Bluetooth, QR code scanning, or tap-to-display. These methods have been described in detail above, and will not be repeated in this embodiment.

[0119] 802. Mobile phone B collects the first audio stream based on its own microphone and sends the first audio stream to mobile phone A.

[0120] During multi-screen collaboration between phone A and phone B, phone A can capture an audio stream (first audio stream) using phone B's microphone. That is, during multi-screen collaboration, phone B can continuously capture an audio stream (first audio stream) using its own microphone and send the audio stream to phone A.

[0121] In some embodiments, the multi-screen collaboration scenario includes a collaborative calling scenario. Collaborative calling includes Voice over Internet Protocol (VoIP) calls and / or SIM card calls. VoIP can also be referred to as broadband telephony or IP telephony.

[0122] For example, in a collaborative call scenario, the master device (phone A) can collect audio data (sound pickup) using the microphone of the controlled device (phone B). The controlled device (phone B) can then send the audio data collected by the microphone to the master device (phone A), which in turn sends the audio data collected by the microphone to the remote device communicating with the master device. The call between the master device and the remote device can be a VoIP call or a SIM card call.

[0123] In some embodiments, multi-screen collaboration scenarios include collaborative audio recording scenarios and / or collaborative speech recognition scenarios.

[0124] For example, in a collaborative audio recording scenario, the master device (phone A) can collect audio data (sound pickup) based on the microphone of the controlled device (phone B), the controlled device (phone B) can send the audio data collected by the microphone to the master device (phone A), and the master device (phone A) can record the audio data collected by the microphone of the controlled device (phone B).

[0125] Collaborative speech recognition scenarios can include text-to-speech conversion scenarios and intelligent voice assistant scenarios.

[0126] For example, in a text-to-speech conversion scenario, the master device (phone A) can collect audio data (sound pickup) using the microphone of the controlled device (phone B). The controlled device (phone B) can then send the audio data collected by its microphone to the master device (phone A). The master device (phone A) can then perform text-to-speech conversion based on the audio data collected by the microphone of the controlled device (phone B), that is, convert the audio data collected by the microphone of the controlled device (phone B) into text. Alternatively, the controlled device (phone B) can convert the audio data collected by its microphone into text and then send it to the master device (phone A).

[0127] For example, in a smart voice assistant scenario, the master device (phone A) can collect audio data (audio commands) from the microphone of the controlled device (phone B). The controlled device (phone B) can then send the audio commands collected by its microphone to the master device (phone A). The master device (phone A) can then perform corresponding operations based on the audio commands collected by the microphone of the controlled device (phone B). For instance, assuming the audio command collected by the microphone of the controlled device (phone B) is to open a video application, the master device (phone A) can open the video application. Alternatively, the controlled device (phone B) can perform corresponding operations based on the audio data (audio commands) collected by its microphone. For instance, assuming the audio command collected by the microphone of the controlled device (phone B) is to open a video application, the controlled device (phone B) can open the video application.

[0128] 803. Mobile phone B monitors the call status of its own SIM card or eSIM card.

[0129] During the multi-screen collaboration between mobile phone A and mobile phone B, mobile phone B can continuously monitor the call status of its own SIM card or eSIM card.

[0130] In some embodiments, mobile phone B may include one or more SIM cards or eSIM cards, and the mobile phone may monitor the call status of all its SIM cards or eSIM cards.

[0131] 804a. Mobile phone B receives a call from a remote electronic device (e.g., mobile phone C).

[0132] A remote electronic device (a third electronic device, such as mobile phone C) can be the calling device, and mobile phone B can be the called device. The remote electronic device can make calls to mobile phone B using its own SIM card or eSIM card. Mobile phone B can receive calls from the remote electronic device using its own SIM card or eSIM card.

[0133] 804b. When the SIM card or eSIM card of mobile phone B is in the answering state, mobile phone B stops collecting the first audio stream.

[0134] When a user answers a call from a remote electronic device, the SIM card or eSIM card of mobile phone B is in the answering state. At this time, mobile phone B stops collecting the first audio stream, that is, it stops collecting audio streams for the multi-screen collaboration service.

[0135] In some embodiments, mobile phone B may display a first pop-up window, which is used to prompt the user that the audio stream in the multi-screen collaboration scenario has been disconnected.

[0136] 805. Mobile phone B acquires a second audio stream based on its microphone and transmits the second audio stream to a remote electronic device (e.g., mobile phone C) based on a SIM card or eSIM card.

[0137] After mobile phone B stops collecting audio streams for multi-screen collaboration services, it can collect audio streams (second audio streams) for operator call services based on the microphone, and can send the second audio streams to remote electronic devices based on SIM cards or eSIM cards to ensure normal communication between mobile phone B and remote electronic devices.

[0138] In some embodiments, mobile phone B can send a second audio stream to a base station based on a SIM card or eSIM card, and then send the second audio stream to a third electronic device via the base station.

[0139] Based on the method provided in this application, during multi-screen collaboration between mobile phone A and mobile phone B, mobile phone B can monitor the call status of its own SIM card or eSIM card. When the call status of mobile phone B's SIM card or eSIM card is in an answering state, mobile phone B can stop collecting the first audio stream, i.e., stop collecting audio streams for the multi-screen collaboration service, and instead collect audio streams (second audio streams) for the carrier call service using its microphone, thus ensuring normal communication between mobile phone B and the remote electronic device. This avoids the problem of the collaboration service and the carrier call service simultaneously using the microphone, which could lead to mutual interference and a degraded user experience.

[0140] It should be noted that when the multi-screen collaboration scenario is a collaborative call scenario, the method provided in the embodiments of this application can avoid the collaborative call and the carrier call of the controlled device (e.g., mobile phone B) from interfering with each other. This can prevent the other end of the collaborative call (the other end of the call with the master device) and the other end of the carrier call (the other end of the call with the controlled device) from being interfered with by irrelevant call content, while ensuring the security of different calls.

[0141] When the multi-screen collaboration scenario involves collaborative audio recording, the method provided in this application not only prevents the controlled device (phone B) from sending audio data of the operator's call content to the master device (phone A), thus preventing the leakage of the operator's call content from the controlled device and ensuring the security of the operator's call, but also prevents the controlled device (phone B) from sending audio data of the operator's call content to the master device (phone A), thus preventing the operator's call content from interfering with the recording content of the master device. Therefore, the method provided in this application can simultaneously guarantee the user experience of both the master device and the controlled device.

[0142] When the multi-screen collaboration scenario involves collaborative voice recognition, the method provided in this application not only prevents the leakage of the carrier call content of the controlled device, ensuring the security of the carrier call, but also prevents the controlled device (phone B) from sending the audio data of the carrier call content to the master device (phone A), thus avoiding interference between the carrier call content of the controlled device and the voice recognition content of the master device.

[0143] For example, in the case of a collaborative speech recognition scenario involving an intelligent voice assistant, the controlled device (phone B) does not send the audio data of the operator's call content to the master device (phone A). This can prevent the master device or the controlled device from misinterpreting the operator's call content as audio commands to control the intelligent voice assistant, which could lead to misoperation.

[0144] like Figure 9 As shown in the figure, this application provides an audio stream processing method, which further explains the processing flow of mobile phone B after establishing multi-screen collaboration between mobile phone B and mobile phone A, including:

[0145] 901a. In response to the user's first action, the multi-screen collaboration application starts the DMSDP process.

[0146] The first operation is used to trigger multi-screen collaboration. That is, after receiving the operation to trigger multi-screen collaboration, the multi-screen collaboration application can start (launch) the DMSDP process in the application framework layer.

[0147] 901b, DMSDP process initializes the virtual audio proxy module.

[0148] After the DMSDP process starts, it can initialize the VirtualAudioProxy thread to create the audio record needed for sound pickup and the audio track needed for sound playback.

[0149] 901c, The virtual audio agent thread includes an audio state adjustment module, which initializes the microphone's state to a first state (e.g., an available state).

[0150] The first state can indicate that the microphone is available. For example, the first state can be true.

[0151] 901d. When the microphone is in the first state, the virtual audio proxy module acquires the first audio stream based on the microphone.

[0152] For example, when the microphone availability status is true, the virtual audio agent thread can call the audiorecord's read method to acquire a first audio stream based on the microphone.

[0153] 902. The DMSDP process creates a call status monitoring module, which monitors the call status of the SIM card or eSIM card of the first electronic device.

[0154] The call status monitoring thread can monitor incoming calls from mobile phone B, including calls from mobile phone B's SIM card or eSIM card.

[0155] In this embodiment, the call state listening thread can be a PhoneStateListener thread. PhoneStateListener is a native listener of the Android system, which does not introduce any additional third-party dependencies and does not add any extra overhead. Furthermore, the call state listening thread is an independent thread in the DMSDP process and will not affect the main thread of the DMSDP process (e.g., it will not block the main thread).

[0156] Understandably, since monitoring the call status of a SIM card or eSIM card needs to be continuous, a looper object can be created for the call status monitoring thread. This looper object allows the call status monitoring thread to continuously monitor the call status of the corresponding SIM card or eSIM card.

[0157] The method provided in this application embodiment can create a call monitoring thread after the collaborative service is started. Based on the call monitoring thread, the call status of the SIM card or eSIM card of the first electronic device can be determined. When the call status of the SIM card or eSIM card of the first electronic device is in an answering state, the acquisition of the first audio stream can be stopped, thereby avoiding the problem of conflict between the multi-screen collaborative service and the use of the microphone by the operator's call of the first electronic device.

[0158] In some embodiments, mobile phone B may include multiple SIM cards and / or eSIM cards. In this case, mobile phone B can create multiple call status monitoring modules, each used to monitor the call status of one SIM card or eSIM card. For example, if mobile phone B includes one SIM card and one eSIM card, mobile phone B can create two call status monitoring modules, one for monitoring the call status of the SIM card and the other for monitoring the call status of the eSIM card.

[0159] 903. When the call status monitoring module detects that the call status is in the answering state, it notifies the audio status management module that the current call status is in the answering state.

[0160] After receiving an incoming call, mobile phone B responds to the user's answering action. The call status monitoring module detects that the call status is in the answering state (e.g., offhook state) and can send a first notification message to the audio status management module.

[0161] The first notification message is used to indicate that the call status is in the answering state, or the first notification message is used to instruct the audio status management module to refresh the microphone status flag to the first flag (e.g., unavailable state).

[0162] It is understandable that when two or more applications (e.g., multi-screen collaboration applications and calling applications) need to use the microphone, microphone usage conflicts may occur. In this embodiment, microphone usage rights can be preempted based on service priority (or application priority), with higher-priority services (or applications) preempting lower-priority services (or applications) from using the microphone.

[0163] In some embodiments, the calling application (the application that makes or receives calls based on a SIM card or eSIM card) has a higher priority than the multi-screen collaboration application. That is, the calling application can preempt the multi-screen collaboration application's microphone access. Once the calling application preempts the multi-screen collaboration application's microphone access, the microphone access belongs exclusively to the calling application, and the multi-screen collaboration application can no longer use the microphone until the call ends, at which point the calling application releases the microphone access.

[0164] For example, if the multi-screen collaboration application starts running first, it can obtain microphone access first. Then, if the call application starts running, it can preempt the multi-screen collaboration application's microphone access because the call application has a higher priority. However, if the call application starts running first, and then the multi-screen collaboration application starts running, it cannot preempt the call application's microphone access because the multi-screen collaboration application has a lower priority. Only after the call application releases its microphone access can the multi-screen collaboration application obtain it.

[0165] 904. The audio status management module sets the microphone status flag to unavailable.

[0166] After receiving the first notification message from the call status monitoring module, the audio status management module can mark the microphone status as a second status (e.g., false). The second status is used to indicate that the microphone is unavailable.

[0167] 905. When the virtual audio proxy thread determines that the microphone status is marked as unavailable, it stops picking up sound.

[0168] When the virtual audio agent thread reads that the microphone status is marked as unavailable, it can stop calling the audio record's read method to capture the audio stream (the first audio stream), that is, stop using the microphone to pick up sound.

[0169] 906a. The virtual audio proxy thread notifies multi-screen collaboration applications that the microphone is unavailable via callback.

[0170] The virtual audio proxy thread can send a first callback status code to the multi-screen collaboration application, which indicates that the microphone is unavailable.

[0171] In this embodiment, the virtual audio proxy thread can agree on one or more callback status codes with the multi-screen collaboration application, with different callback status codes representing different meanings. For example, the multiple callback status codes include a first callback status code and a second callback status code. The first callback status code indicates that the microphone is unavailable. The second callback status code indicates that the microphone is available.

[0172] 906b. Multi-screen collaboration applications display the first prompt message, indicating that the microphone is unavailable to the user.

[0173] The multi-screen collaboration application can display a first pop-up window, which includes a first prompt message to inform the user that the microphone is unavailable.

[0174] For example, such as Figure 10 As shown, the first pop-up can be pop-up 1001, and the first prompt message can be "Other applications are recording (or using the microphone), so 'Multi-screen Collaboration' cannot record (or cannot use the microphone)".

[0175] 907. After a call on mobile phone B is disconnected, the call status monitoring module detects that the call status is disconnected and notifies the audio status management module.

[0176] After the call on mobile phone B is disconnected, the call status monitoring module detects that the call status is disconnected (idle) and can send a second notification message to the audio status management module.

[0177] The second notification message is used to indicate that the call status is hung up, or the second notification message is used to instruct the audio status management module to refresh the microphone status flag to the second flag (e.g., available status).

[0178] 908. The audio status management module sets the microphone status flag to available.

[0179] After receiving the second notification message from the call status monitoring module, the audio status management module can refresh the microphone status flag to an available state (e.g., true).

[0180] 909. The virtual audio proxy thread continues to pick up sound when the microphone status is marked as available.

[0181] When the virtual audio agent thread reads that the microphone status is marked as available, it can call the audiorecord's read method to capture the audio stream (the first audio stream), meaning it can continue to use the microphone to pick up sound.

[0182] 910. The virtual audio proxy thread notifies multi-screen collaboration applications that the microphone is available via callback.

[0183] The virtual audio proxy thread can send a second callback status code to multi-screen collaboration applications, which indicates that the microphone is available.

[0184] 911. The multi-screen collaboration application displays a second prompt message.

[0185] The second notification message is used to inform the user that the microphone is available.

[0186] For example, the second prompt message could be "Other applications have stopped recording (or stopped using the microphone), so Multi-Screen Collaboration has started recording (or is using the microphone)".

[0187] Finally, after the multi-screen collaboration scenario ends, the call status monitoring thread can be destroyed. It is understood that after the multi-screen collaboration scenario ends, i.e., after the multi-screen collaboration service is shut down, the conflict scenario no longer exists (the collaboration service and the carrier call service will not exist simultaneously, therefore no conflict will occur). Destroying the call monitoring thread at this time not only prevents the collaboration service and the carrier call service from interfering with each other, but also clears memory and saves power.

[0188] Based on the method provided in this application embodiment, during multi-screen collaboration between mobile phone A and mobile phone B, the call status monitoring thread of mobile phone B can monitor the call status of its own SIM card or eSIM card. When the call status monitoring thread detects that the SIM card or eSIM card is in an answered state, it can promptly notify the audio status management module of the call status of the SIM card or eSIM card. The audio status management module can refresh the microphone status based on the call status of the SIM card or eSIM card (e.g., refresh to an unavailable state). When the virtual audio proxy thread determines that the microphone status is unavailable, it can stop collecting the first audio stream, that is, stop collecting the audio stream for the multi-screen collaboration service. This allows the operator's call service to collect the audio stream (second audio stream) based on the microphone, ensuring normal communication between mobile phone B and the remote electronic device. This avoids the problem of multi-screen collaboration and operator's call services using the microphone simultaneously, which could cause mutual interference and thus degrade the user experience.

[0189] Some embodiments of this application provide a first electronic device, which may include a touchscreen, a memory, and one or more processors. The touchscreen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the first electronic device can perform various functions or steps performed by the main device in the above method embodiments. The structure of the first electronic device can be referred to... Figure 2 The structure of the first electronic device 100 shown.

[0190] This application also provides a chip system (e.g., a system-on-a-chip (SoC)). Figure 11 As shown, the chip system includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 and the interface circuit 1102 are interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of the first electronic device). As another example, the interface circuit 1102 can be used to send signals to other devices (e.g., the processor 1101 or the touchscreen of the first electronic device). Exemplarily, the interface circuit 1102 can read instructions stored in the memory and send those instructions to the processor 1101. When the instructions are executed by the processor 1101, the first electronic device can perform the steps executed by the main device in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application embodiment.

[0191] This application also provides a computer-readable storage medium including computer instructions that, when executed on the first electronic device, cause the first electronic device to perform various functions or steps performed by the main device in the above method embodiments.

[0192] This application also provides a computer program product that, when run on a first electronic device, causes the first electronic device to perform various functions or steps performed by the main device in the above method embodiments.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio stream processing method, characterized in that, Applied to a first electronic device, the first electronic device including a SIM card or an eSIM card, the first electronic device further including a microphone, the method includes: The first electronic device acquires a first audio stream based on the microphone and sends the first audio stream to the second electronic device. The first electronic device and the second electronic device are in a multi-screen collaboration scenario. The first electronic device monitors the call status of the SIM card or the eSIM card of the first electronic device; When the SIM card or eSIM card of the first electronic device is in the call state of being answered, the acquisition of the first audio stream is stopped; The first electronic device acquires a second audio stream based on the microphone and sends the second audio stream to a third electronic device based on the SIM card or the eSIM card.

2. The method according to claim 1, characterized in that, The multi-screen collaboration scenario includes a collaborative call scenario, wherein collaborative calls include VoIP calls and / or SIM card calls.

3. The method according to claim 1, characterized in that, The multi-screen collaboration scenarios include collaborative audio recording scenarios and / or collaborative speech recognition scenarios.

4. The method according to claim 1, characterized in that, After stopping the acquisition of the first audio stream, the method further includes: The first electronic device displays a first pop-up window, which is used to prompt the user that the audio stream in the multi-screen collaboration scenario has been disconnected.

5. The method according to any one of claims 1-4, characterized in that, Before the first electronic device acquires the first audio stream based on the microphone, the method further includes: In response to the user's first operation, the Distributed Device Virtualization (DMSDP) process is started. The DMSDP process initializes the virtual audio proxy module. The virtual audio proxy module includes an audio state adjustment module. The audio state adjustment module initializes the microphone's state to a first state, which is used to indicate that the microphone is available. The first operation is used to trigger multi-screen collaboration. When the microphone is in the first state, the virtual audio proxy module acquires the first audio stream based on the microphone; The DMSDP process creates a call status monitoring module, and monitors the call status of the SIM card or eSIM card of the first electronic device based on the call status monitoring module. When the call status monitoring module detects that the call status of the SIM card or the eSIM card of the first electronic device is in the answering state, it notifies the audio status adjustment module, which marks the microphone status as a second state, which indicates that the microphone is unavailable. When the microphone is in the second state, the virtual audio proxy module stops acquiring the first audio stream.

6. The method according to claim 5, characterized in that, Call services based on the SIM card or the eSIM card have a higher priority than multi-screen collaboration services.

7. The method according to claim 6, characterized in that, The method further includes: When the call status monitoring module detects that the call status of the SIM card or the eSIM card of the first electronic device is in the hung-up state, it notifies the audio status adjustment module, and the audio status adjustment module re-marks the microphone status to the first state. When the microphone is in the first state, the virtual audio proxy module continues to acquire the first audio stream based on the microphone.

8. The method according to any one of claims 1-7, characterized in that, After the multi-screen collaboration scenario ends, the call status monitoring module is destroyed.

9. The method according to any one of claims 1-8, characterized in that, The first electronic device is a mobile phone or a tablet computer.

10. An audio stream processing method, characterized in that, A method applicable to a system comprising a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are in a multi-screen collaborative state, the first electronic device includes a SIM card or an eSIM card, and the first electronic device also includes a microphone, the method comprising: The first electronic device acquires a first audio stream based on the microphone and sends the first audio stream to the second electronic device; The second electronic device receives the first audio stream; The first electronic device monitors the call status of the SIM card or the eSIM card of the first electronic device; When the SIM card or eSIM card of the first electronic device is in the call state of being answered, the acquisition of the first audio stream is stopped; The first electronic device acquires a second audio stream based on the microphone and sends the second audio stream to a third electronic device based on the SIM card or the eSIM card.

11. An audio stream processing system, characterized in that, The system includes a first electronic device and a second electronic device, which are in a multi-screen collaboration state. The first electronic device includes a SIM card or an eSIM card and a microphone. The first electronic device is used to acquire a first audio stream based on the microphone and send the first audio stream to the second electronic device; The second electronic device is used to receive the first audio stream; The first electronic device is also used to monitor the call status of the SIM card or the eSIM card of the first electronic device; The first electronic device is further configured to stop collecting the first audio stream when the call status of the SIM card or the eSIM card is in the answering state; The first electronic device is further configured to acquire a second audio stream based on the microphone, and send the second audio stream to a third electronic device based on the SIM card or the eSIM card.

12. A first electronic device, characterized in that, The first electronic device includes: a wireless communication module, a memory, and one or more processors; the wireless communication module, the memory, and the processors are coupled together. The memory is used to store computer program code, which includes computer instructions; when the computer instructions are executed by the processor, the first electronic device performs the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, Includes computer instructions; When the computer instructions are executed on the first electronic device, the first electronic device causes the first electronic device to perform the method as described in any one of claims 1-10.

14. A chip system, characterized in that, The chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines. The chip system is applied to a first electronic device including a communication module and a memory; the interface circuit is used to receive signals from the memory and send the signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the first electronic device performs the method as described in any one of claims 1-10.

Images