Audio signal acquisition method and device

Abstract

The embodiment of the invention provides an audio signal acquisition method and device, relates to the field of terminals, and can reduce the power consumption of electronic equipment and prolong the endurance time of the electronic equipment. The method comprises the steps that in response to a first operation, a first interface of a first application is displayed, and the first interface comprises a first microphone icon; after the first operation is received, setting the current microphone equipment as first microphone equipment; under the condition that the current microphone equipment is set as the first microphone equipment, the electronic equipment collects an audio signal through a microphone device; in response to the second operation, a second interface of the first application is displayed, the second interface comprises a second microphone icon, and the second microphone icon is different from the first microphone icon; after the second operation is received, setting the current microphone equipment as second microphone equipment; and under the condition that the current microphone equipment is set as the second microphone equipment, the electronic equipment does not collect the audio signal through the microphone device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminals, and in particular to a method and device for collecting audio signals. Background Art

[0002] The electronic device may include a microphone device, which may be used to collect an audio signal (audio stream).

[0003] At present, in some scenarios (for example, conference scenarios), even if the user triggers the mute operation, the microphone device still does not stop collecting the audio stream. Electronic devices will generate power consumption in the process of collecting audio signals, resulting in an increase in the overall power consumption of electronic devices and a reduction in the battery life of electronic devices. Summary of the invention

[0004] The embodiments of the present application provide an audio signal acquisition method and device, which can reduce the power consumption of electronic devices and increase the battery life of electronic devices.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for collecting audio signals is provided, which is applied to an electronic device, wherein a first application is installed on the electronic device and the electronic device includes a microphone device, the method comprising: in response to a first operation, displaying a first interface of the first application, the first interface including a first microphone icon; after receiving the first operation, setting a current microphone device as the first microphone device; in a case where the current microphone device is set to the first microphone device, the electronic device collects audio signals through the microphone device; in response to a second operation, displaying a second interface of the first application, the second interface including a second microphone icon, the second microphone icon being different from the first microphone icon; after receiving the second operation, setting the current microphone device as the second microphone device; in a case where the current microphone device is set to the second microphone device, the electronic device does not collect audio signals through the microphone device.

[0007] Based on the audio signal acquisition method provided in the embodiment of the present application, after receiving a second operation (for example, a mute operation), the currently used microphone device (microphone driver) can be set to a second microphone device (virtual microphone device). The virtual microphone device will not trigger the microphone device to collect the audio stream and process the audio stream, thereby reducing the power consumption of the electronic device and increasing the battery life of the electronic device.

[0008] In a possible implementation, after receiving the second operation, setting the current microphone device as the second microphone device includes: after receiving the second operation, based on the first application being muted, setting the current microphone device as the second microphone device. In some embodiments, whether the first application is muted can be determined based on the first microphone icon, solving the problem that the microphone status (muted / non-muted status) of a closed-source third-party application (first application) is difficult to obtain.

[0009] In a possible implementation, a second application is also installed on the electronic device, and the method further includes: in response to a third operation, displaying a third interface of the second application, the third interface including a third microphone icon; after receiving the third operation, setting the current microphone device to the first microphone device. After receiving the third operation, the current microphone device can be set to the first microphone device to avoid the problem that the second application cannot collect the audio stream, thereby reducing the user experience.

[0010] In a possible implementation, after receiving the third operation, setting the current microphone device as the first microphone device includes: after receiving the third operation, based on the second application not being muted, setting the current microphone device as the first microphone device. Whether the second application is muted can be determined based on the third microphone icon. If the second application is not muted, the current microphone device can be set to the first microphone device to avoid the problem that the second application cannot collect the audio stream, thereby reducing the user experience.

[0011] In a possible implementation, the method further includes: in response to the fourth operation, closing the audio session of the second application; after receiving the fourth operation, setting the current microphone device to the second microphone device. After the audio session of the second application is closed, there is no need to collect the audio stream, so the current microphone device can be set to the second microphone device, and the second microphone device will not trigger the microphone device to collect the audio stream and process the audio stream, thereby reducing the power consumption of the electronic device.

[0012] In a possible implementation, the electronic device further includes a third application. After receiving the first operation, the method further includes: the third application determines whether the process name of the first application matches a whitelist, and the whitelist includes the process names of one or more preset processes; when the process name of the first application matches the whitelist, the third application takes a screenshot of the first interface to obtain a first screenshot including a first microphone icon, and determines whether the first application is muted based on the first screenshot; when the first application is not muted, a second microphone device is installed and the second microphone device is disabled. In this way, it is possible to determine whether the first application is muted based on the first screenshot (including the first microphone icon), solving the problem that the microphone status (muted / non-muted status) of a closed-source third-party application (first application) is difficult to obtain.

[0013] In a possible implementation, after receiving the second operation, the current microphone device is set to the second microphone device based on the first application being muted, including: after receiving the second operation, the third application takes a screenshot of the second interface to obtain a second screenshot containing the second microphone icon, and determines whether the first application is muted according to the second screenshot; when the first application is muted, the third application enables the second microphone device and sets the current microphone device to the second microphone device. Compared with continuously performing screenshot detection, performing screenshot detection in response to the second operation (i.e., detecting whether the first application is muted according to the first screenshot) can effectively improve detection efficiency and save power consumption.

[0014] In a possible implementation, the electronic device also includes a fourth application, which determines whether the first application is muted based on the first screenshot, including: the third application sends the first screenshot to the fourth application; the fourth application determines whether the first application is muted based on the first screenshot, and returns the determination result to the third application.

[0015] In a possible implementation, determining whether the first application is muted according to the first screenshot includes: inputting the first screenshot into a detection model, obtaining an output result of the detection model, and the output result is used to indicate whether the first application is muted. The detection model can be trained by multiple pictures containing different types of muted icons and non-muted icons.

[0016] In a possible implementation, the second operation includes a mouse operation, a keyboard operation, or a touch screen operation. In a Windows system, the mute state of an application can be set based on the mouse operation, the keyboard operation, or the touch screen operation. Therefore, after detecting the mouse operation, the keyboard operation, or the touch screen operation, a screenshot detection can be performed (i.e., detecting whether the first application is mute based on the first screenshot), which can effectively improve the detection efficiency and save power consumption.

[0017] In a possible implementation, the electronic device includes at least one of a notebook computer, a tablet computer, a portable computer, a desktop computer, and a PDA.

[0018] In a second aspect, the present application provides a chip system, which includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by lines. The above chip system can be applied to an electronic device including a communication module and a memory. The interface circuit is used to receive a signal from the memory of the electronic device and send the received signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device can execute the method described in the first aspect and any possible design thereof.

[0019] In a third aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device (such as a mobile phone), the electronic device executes the method described in the first aspect and any possible design thereof.

[0020] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any possible design thereof.

[0021] In a fifth aspect, an embodiment of the present application provides a process management device for an application, including a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the device implements the method described in the first aspect and any possible design thereof. The device may be an electronic device or a server device; or may be a component of an electronic device or a server device, such as a chip.

[0022] In the sixth aspect, an embodiment of the present application provides a process management device for an application, which can be divided into different logical units or modules according to functions, and each unit or module performs different functions, so that the device executes the method described in the above-mentioned first aspect and any possible design method thereof.

[0023] It can be understood that the beneficial effects that can be achieved by the chip system described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the devices described in the fifth aspect and the sixth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a process flow provided for an embodiment of the present application;

[0027] Figure 4 A schematic diagram of signal interaction between modules provided in an embodiment of the present application Figure 1 ;

[0028] Figure 5 A display diagram provided in the embodiment of the present application Figure 1 ;

[0029] Figure 6 A display diagram provided in the embodiment of the present application Figure 2 ;

[0030] Figure 7 A display diagram provided in the embodiment of the present application Figure 3 ;

[0031] Figure 8 A display diagram provided in the embodiment of the present application Figure 4 ;

[0032] Fig. 9 A display diagram provided in the embodiment of the present application Figure 5 ;

[0033] Fig.10 A display diagram provided in the embodiment of the present application Figure 6 ;

[0034] Fig.11 A display diagram provided in the embodiment of the present application Figure 7 ;

[0035] Fig.12 A schematic diagram of signal interaction between modules provided in an embodiment of the present application Figure 2 ;

[0036] Fig.13 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the description of the following embodiments clear and concise, a brief introduction to the relevant concepts or technologies is first given:

[0038] At present, in some scenarios (for example, conference scenarios), even if the user triggers the mute operation, the microphone device still does not stop collecting the audio stream. Electronic devices will generate power consumption in the process of collecting audio signals, resulting in an increase in the overall power consumption of electronic devices and a reduction in the battery life of electronic devices.

[0039] In one possible design, when the microphone is muted, the electronic device can control the microphone device to stop collecting audio streams through the microphone driver. However, controlling the microphone device through the microphone driver requires cooperation and negotiation between the supplier of the operating system of the electronic device (e.g., PC) (e.g., Microsoft®), the supplier of the microphone driver (e.g., Intel®), the supplier of the microphone device, and the device manufacturer (e.g., Honor), and the process is complicated. And because the PC's operating system (e.g., Windows® operating system) and the microphone driver (e.g., Intel® driver or other third-party drivers) are closed source, it is difficult for the device manufacturer (e.g., Honor) to control the microphone device through the microphone driver (e.g., control the microphone device to stop collecting audio streams). In addition, the source code of third-party applications (e.g., conference applications) is usually closed source, and it is difficult to obtain the microphone status (mute / non-mute status) of the third-party application.

[0040] An embodiment of the present application provides an audio signal acquisition method, which can identify the microphone state (mute / non-mute state) of a third-party application. When the microphone state is identified as a mute state (i.e., the microphone is muted), the currently used microphone device (microphone driver) can be replaced with a virtual microphone device. The virtual microphone device will not trigger the microphone device to collect audio streams (audio signals) and process the audio streams, thereby reducing the power consumption of electronic devices and increasing the battery life of electronic devices.

[0041] The application process management method provided in the embodiment of the present application can be applied to electronic devices. The electronic devices can be, for example, tablet computers, notebook computers (laptop computers), desktop computers (desktop computers), handheld computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPCs), netbooks, personal digital assistants (personal digital assistants, PDAs), mobile phones, etc. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic devices.

[0042] Figure 1 FIG. 1 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone device 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0043] Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0044] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0045] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0046] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture static images or videos. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0047] The camera 193 may include 1 to N cameras. Each camera includes a photosensitive element (CCD / CMOS), which can sense light, collect photons and convert them into electric charges.

[0048] The methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure.

[0049] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, a Windows® operating system (OS) with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0050] In some embodiments, Figure 2 As shown, the Windows® operating system may include a user mode and a kernel mode. It should be noted that the embodiments of the present application are described by taking the Windows® operating system as an example. In other operating systems, as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the solutions of the present application can also be implemented.

[0051] Among them, user mode is the mode for running applications in the Windows operating system. In user mode, applications can access some system resources, such as the file system, network, process, and thread, but cannot directly access kernel mode or underlying hardware devices. User mode can also be used to implement components such as the user interface of the Windows operating system (including the desktop, taskbar, window, menu, and dialog box). These components provide a way for users to interact and operate with the operating system. User mode can include user programs. Among them, user programs can be 32-bit applications or 64-bit applications.

[0052] In an embodiment of the present application, the user mode may include Windows API (application program interface), a first application (e.g., a conference application), a second application (e.g., a call application), a third application (e.g., a housekeeper application), and a fourth application (e.g., a smart middle office application).

[0053] Among them, Windows API can be used to manage microphone devices. The housekeeper application can switch microphone devices through Windows API.

[0054] The conference application allows multiple people to hold online meetings (video conferences or audio conferences).

[0055] The calling application allows users to make voice calls / video calls between each other.

[0056] The housekeeper application can be used to manage the applications installed on the PC, and can perform virus detection, garbage cleaning and other operations.

[0057] In some embodiments, the housekeeper application can take screenshots of the interface of an application (such as a conference application, a call application) and send the screenshots to the smart middle office application.

[0058] The smart middle platform application can be used to determine the state (mute state or non-mute state) of an application (such as a conference application or a call application) based on a screenshot of the application. The smart middle platform application can include a detection model, which can be trained by multiple pictures containing different types of mute icons and non-mute icons.

[0059] In other embodiments, the housekeeper application may take a screenshot of the interface of an application (such as a conference application, a call application), and determine the state of the application (muted state or non-muted state) based on the screenshot.

[0060] Kernel mode can include components such as Windows execution entity, Windows kernel, device drivers and hardware abstraction layer. Among them, Windows execution entity can include basic system services, such as memory manager, process and thread manager, security management, I / O management, network and inter-process communication, etc. Windows kernel can include low-level system functions, such as thread scheduling, interrupt, exception distribution and multi-core synchronization. Windows kernel also provides some routines and basic objects for implementing high-level structures. Device drivers can include hardware device drivers and software drivers (such as file and network drivers). The hardware abstraction layer is a layer of code independent of the kernel, which separates device drivers from platform differences.

[0061] In the embodiment of the present application, the kernel mode device driver may include an audio driver and a virtual audio driver.

[0062] The audio driver may include a microphone driver, which may control the microphone device to collect audio signals.

[0063] A virtual audio driver can "disguise" itself as an audio driver (e.g., a microphone driver). It has no corresponding physical device (microphone device) and cannot drive the microphone device to collect audio signals. A virtual audio driver corresponds to a virtual microphone device. A virtual microphone device cannot collect audio signals.

[0064] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. Wherein, in the description of the present application, unless otherwise specified, "at least one" refers to one or more, and "a plurality of" refers to two or more than two. In addition, in order to facilitate the clear description of the technical solutions in the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit necessarily different.

[0065] For ease of understanding, the audio signal acquisition method provided in the embodiment of the present application is specifically introduced below with reference to the accompanying drawings.

[0066] like Figure 3 As shown, an embodiment of the present application provides an audio signal acquisition method, which is applied to an electronic device, the electronic device is installed with a first application, and the electronic device includes a microphone device, the method comprising:

[0067] 301. In response to a first operation, display a first interface of a first application, where the first interface includes a first microphone icon.

[0068] The first application may be an application program having an audio session function, wherein the audio session function may include functions such as local recording, audio call / video call, etc.

[0069] Exemplarily, the first application may be a conference application. The conference application may be used by multiple people to hold an online conference (video conference or audio conference). The first application (eg, conference application) may be a third-party application, and its source code is usually closed source.

[0070] The first operation may be an operation of joining a conference (e.g., a video conference). For example, a user (e.g., user A) may create / initiate a conference and join it, or a user may join a conference created by another user (e.g., user B), which is not limited in this application. The first interface may be a conference interface, and the conference interface may include a first microphone icon. The first microphone icon may be used to indicate that the current microphone is in a non-muted state (i.e., the first application is not muted).

[0071] 302. After receiving the first operation, set the current microphone device as the first microphone device.

[0072] Specifically, after receiving the first operation, based on the fact that the first application is not muted, the electronic device may set the current microphone device as the first microphone device.

[0073] The fact that the first application is not muted may be determined based on a first microphone icon on the first interface. The first microphone icon may be used to indicate that the current microphone is in a non-muted state (ie, the first application is not muted).

[0074] It should be noted that, when the current microphone device is set as the first microphone device, the electronic device can collect audio signals through the microphone device. The first microphone device is a driver corresponding to the microphone device (for example, an Intel® driver or other third-party driver), which can drive the microphone device to collect audio signals. The source code of the first microphone device is closed source.

[0075] 303. In response to the second operation, display a second interface of the first application, where the second interface includes a second microphone icon, which is different from the first microphone icon.

[0076] The second operation may be a mute operation, for example, the second operation may be a click operation of the user on the first microphone icon. The second microphone icon may be used to indicate that the current microphone is in a mute state (ie, the first application is muted).

[0077] 304. After receiving the second operation, set the current microphone device as the second microphone device.

[0078] After receiving the second operation, based on the first application being muted, the electronic device can set the current microphone device as the second microphone device. The current microphone device refers to the current system default microphone device.

[0079] When the current microphone device is set as the second microphone device, the electronic device does not collect audio signals through the microphone device. The second microphone device is a virtual microphone driver (virtual microphone device) and has no corresponding physical device (microphone device), so the microphone device cannot be driven to collect audio signals. Therefore, when the current microphone device is set as the second microphone device, the electronic device does not collect audio signals through the microphone device.

[0080] Based on the method provided in the embodiment of the present application, the microphone state (mute / non-mute state) of a third-party application (such as the first application) can be identified based on the microphone icon (the first microphone icon and the second microphone icon), which solves the problem that the microphone state (mute / non-mute state) of a closed-source third-party application (the first application) is difficult to obtain. Further, when the microphone state is identified as a mute state (i.e., the microphone is muted), the currently used microphone device (microphone driver) can be replaced with a virtual microphone device, which will not trigger the microphone device to collect audio streams (audio signals) and process audio streams, thereby reducing the power consumption of electronic devices and increasing the battery life of electronic devices. In this way, there is no need to control the microphone device through a native closed-source microphone driver (first microphone device), and there is no need for the supplier (e.g., Intel®) of the microphone driver (first microphone device) of the electronic device (e.g., PC), the supplier of the microphone device and the device (e.g., PC) manufacturer, and the third-party application (e.g., conference application, call application, etc.) manufacturer to cooperate and negotiate, which simplifies the control process and saves manpower, material resources and time costs.

[0081] In some embodiments, a second application is also installed on the electronic device, and the method further includes:

[0082] 305. In response to the third operation, display a third interface of the second application, where the third interface includes a third microphone icon.

[0083] Exemplarily, the second application may be a call application, which may be used for users to make voice calls / video calls. The third operation may be an operation of initiating a voice call (video call) or answering a voice call (video call). The third interface may be an audio / video call interface, which may include a third microphone icon, which may indicate that the current microphone is not muted.

[0084] 306. After receiving the third operation, set the current microphone device as the first microphone device.

[0085] Specifically, after receiving the third operation, based on the fact that the second application is not muted, the current microphone device may be set as the first microphone device.

[0086] The fact that the second application is not muted may be determined based on a third microphone icon on the third interface. The third microphone icon may indicate that the current microphone is in a non-muted state (ie, the second application is not muted).

[0087] When the current microphone device is set as the first microphone device, the electronic device can collect the audio signal through the microphone device. The second application can send the audio signal to the call peer device.

[0088] 307. In response to the fourth operation, close the audio session of the second application.

[0089] The audio session may include recording, audio call / video call, etc. Exemplarily, the fourth operation may be an operation of closing / exiting recording, audio call / video call.

[0090] 308. After receiving the fourth operation, set the current microphone device as the second microphone device.

[0091] Specifically, after receiving the fourth operation, based on the first application being muted, the current microphone device is set to the second microphone device.

[0092] Based on the audio signal acquisition method provided in the embodiment of the present application, when it is recognized that the application is muted, the currently used microphone device (microphone driver) can be set to a second microphone device (virtual microphone device). The virtual microphone device will not trigger the microphone device to collect the audio stream and process the audio stream, thereby reducing the power consumption of the electronic device and increasing the battery life of the electronic device.

[0093] like Figure 4 As shown, the embodiment of the present application provides an audio signal acquisition method, which is described by taking the electronic device as a PC as an example. The PC may include a first application (e.g., a conference application), a second application (e.g., a call application), a third application (e.g., a housekeeper application), a fourth application (e.g., a smart middle station application), a first microphone device, a second microphone device, and a microphone device. The method includes:

[0094] 401a. Start the housekeeper application.

[0095] In some embodiments, in response to a user clicking on the icon of the housekeeper application, the electronic device may start the housekeeper application. Alternatively, the user may set the housekeeper application to start automatically when the electronic device is turned on, so that the housekeeper application is automatically pulled up (started) when the electronic device is turned on.

[0096] 401b. The housekeeper application calls the first interface to monitor the status of the microphone device.

[0097] After the housekeeper application is started, the first interface of the Windows API can be called. The first interface can be used to monitor whether the application has entered the audio session, and the process name corresponding to the application entering the audio session can be reported to the housekeeper application. After the application enters the audio session, the microphone device can be started through the Windows API.

[0098] Exemplarily, the first interface may be OnDefaultDeviceChanged().

[0099] 402a. In response to a first operation (eg, an operation of joining a conference), the conference application displays a first interface, where the first interface includes a first microphone icon.

[0100] Exemplarily, in response to a user's operation on the conference application icon (eg, clicking the conference application icon with a mouse), the PC may start the conference application (ie, start the conference application). Figure 5 As shown, after the conference application is started, the PC may display a window 601, which may include a join conference control 602. The window 601 may also include a join conference control 603. In response to the user's operation on the join conference control 602 (for example, by clicking the join conference control 602 with a mouse) and inputting the conference ID and conference password, or in response to the user's operation on the join conference control 603 (for example, by clicking the join conference control 603 with a mouse), the corresponding conference (for example, XX conference) may be joined.

[0101] For example, Figure 6 As shown, in response to the user's operation on the join conference control 602 and the operation of inputting the conference ID and conference password (an example of the first operation), or in response to the user's operation on the join conference control 603 (another example of the first operation), the PC may display a conference interface 610 (a first interface), and the conference interface 610 may include a microphone icon 611 (an example of the first microphone icon). The microphone icon 611 is used to indicate that the current microphone state is not muted.

[0102] 402b. The conference application sets the current microphone device as the first microphone device.

[0103] After receiving the first operation, the conference application may enter the audio session and set the current microphone device as the first microphone device. The first microphone device may start the microphone device to collect the audio stream.

[0104] 403a. The first interface of the Windows API monitors that the conference application enters the audio session.

[0105] After the first interface monitors that the conference application enters the audio session (ie, the conference application starts the microphone device), step 403b can be executed, that is, the process name (or application name, the embodiment of the present application uses the process name as an example) of the conference application is sent to the butler application.

[0106] 403b. The first interface of the Windows API sends the process name of the conference application to the housekeeper application.

[0107] 404. The housekeeper application determines whether the process name of the conference application matches the whitelist.

[0108] The housekeeper application can determine whether the process name of the conference application matches the process name in the whitelist. That is, determine whether the process name of the conference application exists in the whitelist. The whitelist may include the process names of one or more preset processes. The preset process may include a process related to the audio session, and the process related to the audio session may refer to a process that can call the microphone driver (first microphone device) to control the microphone device to collect audio data.

[0109] Exemplarily, the processes related to the audio session may include the processes of conference applications, the processes of call applications (e.g., telephone applications, chat applications, etc.), the processes of recording applications, etc. That is, the whitelist may include the process names of conference applications, call applications, recording applications, etc.

[0110] In one possible design, the whitelist may not include the browser process name. The scenario of conducting an audio session (eg, joining a meeting or conducting an audio call) through a browser (web page) is an abnormal scenario and is not within the scope of this application.

[0111] If the housekeeper application determines that the process name of the conference application matches the process name in the whitelist, step 405 may be executed. Otherwise, the housekeeper application may wait for the next notification message sent by the first interface to determine whether the process name carried in the next notification message matches the process name in the whitelist.

[0112] In some embodiments, when the housekeeper application determines that the process name of the conference application matches the process name in the whitelist, it can also determine whether the current scene is an audio session based on the command line in the process details. For example, if the command line includes "type=iron", it means that the current scene is an audio session, that is, the user has joined the conference.

[0113] 405. The housekeeper application takes a screenshot of the interface (eg, the first interface) of the conference application to obtain a first screenshot.

[0114] The housekeeper application can take a screenshot of the interface (e.g., the first interface) of the conference application based on preset rules to obtain a first screenshot, and the first screenshot includes a first microphone icon. The preset rule may be to take a screenshot of a preset area in the interface (or window) of the conference application at a preset ratio. That is, the screenshot area is a preset area in the interface of the conference application, and the size (size) of the screenshot relative to the size of the interface of the conference application may be fixed. The preset area may be, for example, the lower left area, the lower center area, the lower right area, etc., which is not specifically limited in this application.

[0115] In some embodiments, different preset applications in the whitelist may be bound to different preset areas, that is, the preset areas corresponding to different preset applications may be different. The preset area corresponding to each preset application may be designed according to the characteristics of the preset application (the position where the microphone icon is displayed by the preset application). In this way, different preset areas can be captured for different applications in a more targeted manner, making the screenshot result more accurate.

[0116] For example, Figure 7 As shown in (a) of FIG. 5 , when the window of the conference application is maximized, the screenshot area may be the bottom area 631 of the interface 630; Figure 7 As shown in (b) in FIG. 1 , when the window of the conference application is restored, the screenshot area may be the bottom area 633 of the interface 632; Figure 7 As shown in (c) , when the window of the conference application is minimized, the screenshot area may be the bottom area 635 of the interface 634 .

[0117] 406. The housekeeper application sends the first screenshot to the smart middle office application.

[0118] The housekeeper application can send the first screenshot to the smart middle platform application based on the interprocess communication (IPC) mechanism.

[0119] 407. The smart middle platform application determines whether the conference application is muted based on the first screenshot, and returns the determination result to the housekeeper application.

[0120] In one possible design, the smart middle platform application can input the first screenshot into the detection model, and determine whether the conference application is muted through the detection model. The detection model can be trained by multiple pictures containing different types of mute icons and non-mute icons.

[0121] The detection result includes two cases, the first case is that the conference application is muted, and the second case is that the conference application is not muted. If the detection result is that the conference application is not muted, step 408 may be performed. If the detection result is that the conference application is muted, step 415 may be performed.

[0122] 408. Based on the fact that the conference application is not muted, the housekeeper application installs a second microphone driver and sets the second microphone device to a disabled state (ie, disables the second microphone device).

[0123] For example, the housekeeper application can disable the second microphone device through SetEndpointVisibility(). After the second microphone device is set to the disabled state, the second microphone device is invisible to the user to avoid user misoperation.

[0124] The current microphone device refers to the microphone driver currently used by the electronic device. For example, Figure 8 As shown in (a) of FIG. 6 , the interface 640 includes a control 641. In response to the operation of clicking the control 641 with a mouse pointer, Figure 8 As shown in (b) in FIG. 6 , a list box 642 may be displayed, and the list box 642 includes all microphone drivers installed in the electronic device, and the current microphone driver is the selected microphone driver 643 (consistent with the system (microphone (6-USB audio driver (AudioDevice), an example of the first microphone device)). Optionally, the list box 642 may also include a speaker driver, etc., which is not limited in the present application.

[0125] When the current microphone device is a first microphone device (eg, 6-USB Audio Device), the electronic device can collect audio signals through the microphone device, wherein the first microphone device is a driver corresponding to the microphone device, and can drive the microphone device to collect audio signals.

[0126] 409. The housekeeper application calls the second interface of the Windows API to monitor the target operation.

[0127] The target operation may include mouse operation, keyboard operation and touch screen operation.

[0128] Exemplarily, the second interface may be SetWindowsHookEx. The second interface may be used to monitor mouse operations, keyboard operations, and touch screen operations.

[0129] In some embodiments, the second interface may monitor mouse operations or touch screen operations on a preset area (eg, the area below the window of the conference application) and keyboard operations on a preset key (eg, a screenshot shortcut key).

[0130] After the second interface monitors the mouse operation, keyboard operation and touch screen operation, it can notify the housekeeper application so that the housekeeper application can promptly detect whether the application program (for example, the conference application) is muted.

[0131] In the embodiment of the present application, step 409 and subsequent steps (such as step 410 to step 414 ) may be executed in parallel with step 404 to step 408 .

[0132] 410. In response to a second operation (eg, a mute operation performed via a mouse, keyboard, or touch screen), the conference application displays a second interface, where the second interface includes a second microphone icon, which is different from the first microphone icon.

[0133] For example, the second operation may be an operation of muting the sound by using a mouse, keyboard or touch screen. Fig. 9 As shown in (a) of FIG. 5 , the interface 650 (an example of the first interface) may include a microphone icon 651 (an example of the first microphone icon). In response to the second operation, Fig. 9 As shown in (b), the PC may display an interface 652 (an example of the second interface), and the interface 652 includes a microphone icon 653 (an example of the second microphone icon). The microphone icon 653 is an icon indicating that the microphone is muted.

[0134] 411a. The second interface of the Windows API monitors a second operation (an operation of muting sound through a mouse, keyboard, or touch screen).

[0135] 411b. The second interface of the Windows API notifies the housekeeper application that the conference application currently receives the second operation (an operation of muting the audio through the mouse, keyboard, or touch screen).

[0136] 412. The housekeeper application takes a screenshot of the second interface of the conference application to obtain a second screenshot.

[0137] Among them, the second screenshot includes a second microphone icon.

[0138] 413. The housekeeper application sends the second screenshot to the smart middle office application.

[0139] 414. The smart middle platform application determines whether the conference application is muted based on the second screenshot, and returns the determination result to the housekeeper application.

[0140] For the description of steps 412 to 414, please refer to steps 405 to 407, which will not be described in detail here.

[0141] It should be noted that, compared with continuously performing screenshot detection, performing screenshot detection in response to the second operation (i.e., detecting whether the conference application is muted based on a screenshot of a preset area of ​​the conference application) can effectively improve detection efficiency and save power consumption.

[0142] If it is determined according to the second screenshot that the conference application is muted, step 415 may be executed.

[0143] 415. Based on the conference application being muted, the housekeeper application sets the second microphone device to an enabled state, and sets the current microphone device as the second microphone device.

[0144] Exemplarily, the housekeeper application can enable the second microphone device through SetEndpointVisibility(), that is, set the second microphone device to an enabled state. The housekeeper application can set the current microphone device to the second microphone device through SetDefaultEndpoint().

[0145] The second microphone device is a virtual microphone driver, and has no corresponding physical device (microphone device), and cannot drive the microphone device to collect audio signals. Therefore, when the current microphone device is set as the second microphone device, the electronic device does not collect audio signals through the microphone device.

[0146] After the second microphone device is set to the enabled state, the second microphone device is visible to the user. Fig.10 As shown in (a) in FIG. 1 , in response to the operation of clicking the control 641 with the mouse pointer, as shown in FIG. Fig.10 As shown in (b) in FIG. 6 , a list box 642 may be displayed, which includes all microphone drivers installed in the electronic device, and the current microphone driver (system default microphone driver) is the selected microphone driver 644 (honor audio driver, an example of the second microphone device). Optionally, the list box 642 may also include a speaker driver, etc., which is not limited in this application.

[0147] In this way, when the conference application is muted, the electronic device does not collect audio signals through the microphone device, which can reduce the power consumption of the electronic device and increase the battery life of the electronic device.

[0148] Furthermore, the embodiment of the present application may also include the following steps:

[0149] 416a. In response to the operation of closing the audio session of the conference application, the conference application closes the related interface (eg, the first interface or the second interface) of the audio session.

[0150] For example, Fig.11 As shown in (a) of FIG. 5 , the interface 650 (an example of the first interface) may include a control 661. In response to a user operation on the control 661 (eg, an operation of clicking the control 661 with a mouse), as shown in FIG. Fig.11As shown in (b) of FIG. 6 , a pop-up box 662 may be displayed, and the pop-up box 662 may include multiple options, such as leaving the meeting, ending the meeting, and canceling. In response to the user's operation on the leaving meeting option or the ending meeting option (for example, by clicking the leaving meeting option or the ending meeting option with a mouse), the conference application may exit the audio session and close the interface 650.

[0151] 416b. The first interface of the Windows API monitors that the conference application exits the audio session.

[0152] The first interface may also be used to monitor whether the application program exits the audio session, and may report the process name corresponding to the application program that exits the audio session to the housekeeper application.

[0153] 416c. The first interface of the Windows API notifies the housekeeper application that the conference application exits the audio session.

[0154] 416d. The housekeeper application disables the second microphone device and sets the current microphone device as the first microphone device.

[0155] The housekeeper application disables the second microphone device, that is, the housekeeper application sets the second microphone device to a disabled state. In addition, the housekeeper application can set the current microphone device to the first microphone device (for example, the housekeeper application can set the current microphone device to the first microphone device through SetDefaultEndpoint()). In this way, it can be avoided that the next audio session cannot use the microphone device to collect audio streams.

[0156] In some embodiments, when the housekeeper application determines that the current scene is an audio session, the identifier corresponding to the current audio session and the sound state (mute state or non-mute state) corresponding to the current audio session can be stored in the audio session list. That is, the identifier of the currently started audio session and the sound state corresponding to the audio session can be saved in the audio session list. Among them, the identifier corresponding to the audio session can be the application name or process name or session identifier corresponding to the audio session, which is not limited here. Subsequently, when starting a new audio session or closing an already started audio session, the housekeeper application can update the audio session list. For example, when adding a new audio session, the housekeeper application can store the identifier corresponding to the new audio session and its corresponding sound state in the audio session list. When closing an audio session, the identifier corresponding to the closed audio session and its sound state can be deleted from the audio session list. Further, the housekeeper application can set the current microphone device according to the updated audio session list. When all the remaining audio sessions in the updated audio session list are in a mute state, the housekeeper application can set the current microphone device to the second microphone device. In this way, the electronic device does not collect audio signals through the microphone device, which can save power consumption. If there is an audio session in a non-muted state among all the remaining audio sessions in the updated audio session list, the current microphone device can be set as the first microphone device, so as to avoid the problem that the audio session in a non-muted state cannot use the microphone device to collect the audio stream.

[0157] The following takes the case where the audio session list includes an audio session corresponding to the first application (the identifier corresponding to the audio session may be 1, and the sound state corresponding to the audio session may be a mute state) as an example to illustrate a case where the PC starts a new audio session and closes an existing (started) audio session.

[0158] like Fig.12 As shown, after step 415 and before step 416a, the method provided in the embodiment of the present application may further include:

[0159] 420. In response to the third operation, display a third interface of the second application (eg, a call application), where the third interface includes a third microphone icon.

[0160] Exemplarily, the third operation may be an operation of starting an audio session (eg, initiating an audio call or answering an audio call). The third interface may be an audio / video call interface, which may include a third microphone icon, which may indicate that the current microphone is not muted.

[0161] Based on the audio session of the second application being started, the housekeeper application can store the identifier (e.g., 2) corresponding to the audio session of the second application and the sound state (e.g., non-mute state) corresponding to the audio session in the audio session list. In this case, the audio session list can be as shown in Table 1:

[0162] Table 1

[0163] The identifier of the audio session The sound status of the audio session 1 (corresponding to the first application) Mute state 2 (corresponding to the second application) Not mute

[0164] 421a. The first interface of the Windows API monitors that the call application enters the audio session.

[0165] After the first interface monitors that the call application starts the audio session, step 421b may be executed, that is, the process name of the conference application is sent to the housekeeper application.

[0166] 421b. The first interface of the Windows API sends the process name of the call application to the housekeeper application.

[0167] 422. The housekeeper application determines whether the process name of the call application matches the whitelist.

[0168] 423. The housekeeper application takes a screenshot of the window of the call application to obtain a third screenshot.

[0169] Among them, the third screenshot includes a third microphone icon.

[0170] 424. The housekeeper application sends the third screenshot to the smart middle office application.

[0171] 425. The smart middle platform application determines whether the call application is muted based on the third screenshot, and returns the determination result to the housekeeper application.

[0172] For steps 422-425, reference may be made to the relevant descriptions of steps 404-407 above, which will not be elaborated here.

[0173] 426. Based on the fact that the call application is not muted, the housekeeper application sets the current microphone device as the first microphone device.

[0174] That is, after receiving the third operation, based on the fact that the second application is not muted, the housekeeper application can set the current microphone device to the first microphone device (i.e. switch from the second microphone device to the first microphone device). When the current microphone device is set to the first microphone device, the microphone device can be used to collect audio signals so that the second application can make an audio call normally.

[0175] After step 426, the following steps may also be included:

[0176] 427a. In response to the fourth operation of closing the audio session of the call application, the call application closes the related interface of the audio session.

[0177] 427b. The first interface of the Windows API monitors that the call application exits the audio session.

[0178] The first interface may also be used to monitor whether the application program exits the audio session, and may report the process name corresponding to the application program that exits the audio session to the housekeeper application.

[0179] 427c. The first interface of the Windows API notifies the housekeeper application that the call application exits the audio session.

[0180] 428. The housekeeper application sets the current microphone device as the second microphone device.

[0181] Exemplarily, the fourth operation may be an operation of closing the audio session of the second application. Based on the closing of the audio session of the second application, the housekeeper application may delete the audio session of the second application from the audio session list, and query the remaining audio sessions and their mute status in the deleted audio session list. In this case, the audio session list may be as shown in Table 2:

[0182] Table 2

[0183] The identifier of the audio session The sound status of the audio session 1 Mute state

[0184] When all remaining audio sessions in the audio session list (e.g., Table 2) are in a muted state, the housekeeper application can set the current microphone device to the second microphone device (i.e., switch from the first microphone device to the second microphone device). In this way, the electronic device does not collect audio signals through the microphone device, which can save power consumption.

[0185] In addition, if there is a non-muted audio session in all the remaining audio sessions in the audio session list, the housekeeper application may not switch the microphone device, that is, keep the current microphone device as the first microphone device. In this way, the problem that the non-muted audio session cannot use the microphone device to collect the audio stream can be avoided.

[0186] It should be noted that the above execution order is only an exemplary description, and this embodiment does not specifically limit the execution order between the steps.

[0187] Based on the audio signal acquisition method provided in the embodiment of the present application, when it is recognized that the application is muted, the currently used microphone device (microphone driver) can be set to a second microphone device (virtual microphone device). The virtual microphone device will not trigger the microphone device to collect the audio stream and process the audio stream, thereby reducing the power consumption of the electronic device and increasing the battery life of the electronic device.

[0188] Some embodiments of the present application provide an electronic device, which may include: a touch screen, a memory, and one or more processors. The touch screen, the memory, and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor runs the computer instructions, the electronic device can perform the various functions or steps performed by the electronic device in the above method embodiment. The structure of the electronic device can refer to Figure 1 The structure of the electronic device 100 is shown.

[0189] The present application also provides a chip system (eg, a system on a chip (SoC)), such as Fig.13 As shown, the chip system includes at least one processor 1301 and at least one interface circuit 1302. The processor 1301 and the interface circuit 1302 can be interconnected through lines. For example, the interface circuit 1302 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 1302 can be used to send signals to other devices (such as the processor 1301 or a touch screen of an electronic device). Exemplarily, the interface circuit 1302 can read instructions stored in the memory and send the instructions to the processor 1301. When the instructions are executed by the processor 1301, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0190] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the electronic device (for example, a mobile phone) in the above-mentioned method embodiment.

[0191] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes each function or step executed by the electronic device (eg, a mobile phone) in the above method embodiment.

[0192] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0197] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for collecting audio signals, characterized in that: Applied to an electronic device, the electronic device has a first application installed thereon, and the electronic device includes a microphone device, the method comprising: In response to a first operation, displaying a first interface of the first application, wherein the first interface includes a first microphone icon; After receiving the first operation, setting the current microphone device as the first microphone device; in the case where the current microphone device is set as the first microphone device, the electronic device collects audio signals through the microphone device; In response to a second operation, displaying a second interface of the first application, the second interface including a second microphone icon, the second microphone icon being different from the first microphone icon; After receiving the second operation, the current microphone device is set as the second microphone device; when the current microphone device is set as the second microphone device, the electronic device does not collect audio signals through the microphone device.

2. The method according to claim 1, characterized in that After receiving the second operation, setting the current microphone device as the second microphone device comprises: After receiving the second operation, based on the first application being muted, setting the current microphone device to the second microphone device.

3. The method according to claim 1 or 2, characterized in that: The electronic device also has a second application installed thereon, and the method further includes: In response to a third operation, displaying a third interface of the second application, wherein the third interface includes a third microphone icon; After receiving the third operation, the current microphone device is set as the first microphone device.

4. The method according to claim 3, characterized in that After receiving the third operation, setting the current microphone device to the first microphone device includes: After receiving the third operation, based on the second application not being muted, setting a current microphone device to the first microphone device.

5. The method according to claim 4, characterized in that The method further comprises: In response to a fourth operation, closing the audio session of the second application; After receiving the fourth operation, the current microphone device is set as the second microphone device.

6. The method according to claim 1 or 2, characterized in that: The electronic device further includes a third application. After receiving the first operation, the method further includes: The third application determines whether the process name of the first application matches a whitelist, where the whitelist includes process names of one or more preset processes; When the process name of the first application matches the whitelist, the third application takes a screenshot of the first interface to obtain a first screenshot including the first microphone icon, and determines whether the first application is muted according to the first screenshot; In a case where the first application is not muted, the second microphone device is installed and the second microphone device is disabled.

7. The method according to claim 6, characterized in that The step of, after receiving the second operation, setting the current microphone device to the second microphone device based on that the first application is muted, comprises: After receiving the second operation, the third application takes a screenshot of the second interface to obtain a second screenshot including the second microphone icon, and determines whether the first application is muted according to the second screenshot; In a case where the first application is muted, the third application enables the second microphone device and sets the current microphone device as the second microphone device.

8. The method according to claim 6, characterized in that The electronic device further includes a fourth application, and the determining whether the first application is muted according to the first screenshot includes: The third application sends the first screenshot to the fourth application; The fourth application determines whether the first application is muted according to the first screenshot, and returns the determination result to the third application.

9. The method according to claim 8, characterized in that The determining, according to the first screenshot, whether the first application is muted includes: The first screenshot is input into a detection model to obtain an output result of the detection model, where the output result is used to indicate whether the first application is muted.

10. The method according to claim 1 or 2, characterized in that: The second operation includes mouse operation, keyboard operation or touch screen operation.

11. The method according to claim 1 or 2, characterized in that: The electronic device includes at least one of a notebook computer, a tablet computer, a portable computer, a desktop computer, and a PDA.

12. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to run a computer program stored in a memory, so that the electronic device implements the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a processor, the method according to any one of claims 1 to 11 is implemented.

Images