Microphone exception processing method, device, apparatus and medium

Abstract

This application relates to the field of microphone technology, and discloses a method, apparatus, device, and medium for handling microphone anomalies. The method includes: acquiring audio sampling signals obtained from multiple microphones in a microphone array; evaluating target features of the audio sampling signals to obtain feature evaluation results; identifying abnormal microphones with signal anomalies based on the feature evaluation results; reconstructing the audio signal at the location of the abnormal microphone based on the audio sampling signals of non-abnormal microphones and the microphone's location to obtain a reconstructed audio signal; and inputting the audio sampling signals of the non-abnormal microphones and the reconstructed audio signal into a voice interaction device to enable the voice interaction device to perform corresponding voice interaction operations. Embodiments of this application can adaptively handle microphone anomalies in a microphone array to improve the voice interaction experience.

Description

Technical Field

[0001] This application relates to the field of microphone technology, and in particular to a microphone anomaly handling method, apparatus, device, and medium. Background Technology

[0002] Currently, smart TV boxes and ODM smart terminals commonly integrate voice assistants. To achieve far-field voice interaction and improve voice recognition accuracy, these devices are typically equipped with multi-microphone arrays. Through collaborative operation, multi-microphone arrays can effectively perform sound source localization, beamforming, noise reduction, and echo cancellation, thereby providing clear voice input in complex environments.

[0003] However, during long-term use, one or more microphones in the microphone array may malfunction for various reasons, such as physical damage, blockage of the microphone aperture by dust or foreign objects, loose internal connections, circuit board failure, or strong electromagnetic interference, resulting in abnormal sound reception (such as excessive noise or sound distortion) or complete failure (no sound). When such malfunctions occur, if they are not detected and adjusted in time, the erroneous or low-quality data provided by the faulty microphones will seriously affect the performance of the entire microphone array, leading to a decrease in voice wake-up rate, an increase in false wake-up rate, a decrease in voice recognition accuracy, and even making the device unable to respond to voice commands, thereby greatly damaging the user experience and the reliability of the device. Summary of the Invention

[0004] The purpose of this application is to provide a microphone anomaly handling method, apparatus, device, and medium that can adaptively handle microphone anomalies in a microphone array to improve the voice interaction experience.

[0005] This application provides a microphone anomaly handling method, including: Acquire audio sampling signals obtained from multiple microphones in a microphone array; The target features of the audio sampled signal are evaluated to obtain feature evaluation results; the target features include the signal power features, signal energy features, signal-to-noise ratio features, cross-correlation features, coherence features, and / or spectral features of the audio sampled signal; Based on the aforementioned feature evaluation results, an abnormal microphone with signal anomalies was identified; Based on the audio sampling signal of the non-abnormal microphone and the location of the microphone, the audio signal at the location of the abnormal microphone is reconstructed to obtain the audio reconstructed signal. The audio sampling signal from the non-abnormal microphone and the audio reconstruction signal are input into the voice interaction device so that the voice interaction device can perform corresponding voice interaction operations.

[0006] In some embodiments, evaluating the target features of the audio sample signal includes: Determine the feature deviation between the target feature and the reference feature of the audio sampling signal; the reference feature is the average value of the target features of other audio sampling signals or a preset target feature threshold. Determine whether the feature deviation value meets the preset feature deviation condition; If it does not meet the requirements, generate a feature evaluation result for the microphone anomaly; If the criteria are met, a feature evaluation result indicating that the microphone is not abnormal is generated.

[0007] In some embodiments, the signal reconstruction of the audio signal at the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone and the location of the microphone includes: Based on the location of the non-abnormal microphones, reconstruct the effective geometric topology of the microphone array; Based on the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone, a signal reconstruction model corresponding to the location of the abnormal microphone is determined; the signal reconstruction model is used to characterize the mapping relationship between the audio sampling signal of the non-abnormal microphone and the expected audio signal at the location of the abnormal microphone. Based on the signal reconstruction model, the audio sampling signal of the non-abnormal microphone is processed to generate an audio reconstruction signal corresponding to the location of the abnormal microphone.

[0008] In some embodiments, the signal reconstruction model is a model based on spatial sound field interpolation, and the processing of the audio sampling signal from the non-abnormal microphone based on the signal reconstruction model includes: Based on the audio sampling signals of the non-abnormal microphones and the effective geometric topology, calculate the spatial correlation characteristics between the non-abnormal microphones; Based on the spatial correlation characteristics and the location of the abnormal microphone, the sound field characteristics at the location of the abnormal microphone are estimated by the signal reconstruction model. The audio reconstruction signal is generated based on the sound field characteristics at the location of the abnormal microphone.

[0009] In some embodiments, the signal reconstruction model is a pre-trained neural network model, and the processing of the audio sampling signal from the non-abnormal microphone based on the signal reconstruction model includes: The audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone are input into the signal reconstruction model; the signal reconstruction model is trained based on the audio sample signal, sample geometric topology, and sample abnormal location. Obtain the audio reconstruction signal output by the signal reconstruction model.

[0010] In some embodiments, before determining the signal reconstruction model corresponding to the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone, the method further includes: Based on the feature evaluation results, the device performance characteristics of the non-abnormal microphone are estimated; Based on the device performance characteristics of the non-abnormal microphone, signal compensation is performed on the audio sampling signal of the non-abnormal microphone to obtain the compensated audio sampling signal of the non-abnormal microphone.

[0011] In some embodiments, after determining the presence of the abnormal microphone, one or more of the following steps are further included: An abnormal prompt message is generated, the pickup distance of the non-abnormal microphone is shortened, and the voice interaction device is switched to near-field interaction mode according to the number of abnormal microphones. Switch to noise reduction operation on the audio sample signal of a single non-abnormal microphone; The voice interaction device is restricted from performing preset target voice interaction operations.

[0012] This application embodiment also provides a microphone anomaly handling device, including: The first module is used to acquire audio sampling signals obtained from multiple microphones in the microphone array; The second module is used to evaluate the target features of the audio sampling signal and obtain the feature evaluation results; the target features include the signal power features, signal energy features, signal-to-noise ratio features, cross-correlation features, coherence features and / or spectral features of the audio sampling signal; The third module is used to identify abnormal microphones with signal anomalies based on the feature evaluation results. The fourth module is used to reconstruct the audio signal at the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone and the location of the microphone, so as to obtain the audio reconstruction signal. The fifth module is used to input the audio sampling signal from the non-abnormal microphone and the audio reconstruction signal into the voice interaction device so that the voice interaction device can perform corresponding voice interaction operations.

[0013] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the microphone abnormality handling method described above.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the microphone anomaly handling method described above.

[0015] The beneficial effects of this application are as follows: By evaluating the target features of the audio sampling signals of each microphone in the microphone array, abnormal microphones can be identified in a timely and accurate manner. Using the audio sampling signals of non-abnormal microphones and the position information of each microphone, the audio signal at the location of the abnormal microphone is reconstructed. Finally, the audio sampling signals of the non-abnormal microphones and the reconstructed audio signal are input into the voice interaction device, ensuring that the voice interaction device can continuously perform the corresponding voice interaction operations. Therefore, by introducing feature evaluation and signal reconstruction mechanisms, the functional deficiencies of abnormal microphones are compensated for, enabling the voice interaction device to receive a complete and high-quality microphone array input. This improves the robustness of the microphone array; even if some microphones fail, the overall system can still maintain stable performance, avoiding damage to the voice interaction function due to microphone failure. It can adaptively handle microphone anomalies in the microphone array, improving the voice interaction experience. Attached Figure Description

[0016] Figure 1 This is an application environment diagram for the microphone anomaly handling method provided in the embodiments of this application.

[0017] Figure 2 This is a flowchart of the microphone anomaly handling method provided in the embodiments of this application.

[0018] Figure 3 This is a flowchart of the specific method of step S102 provided in the embodiments of this application.

[0019] Figure 4 This is a flowchart of the specific method for step S104 provided in the embodiments of this application.

[0020] Figure 5 This is a schematic diagram of the microphone anomaly handling device provided in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0025] The microphone anomaly handling method provided in this application can be executed by a computer device, which can be a terminal device or a server. The terminal device includes, but is not limited to, mobile phones, computers, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed system, or a cloud server. Furthermore, the information, data, and signals involved in this application's embodiments are all authorized by the relevant parties or have received full authorization from all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0026] In intelligent voice interaction devices, microphone array malfunctions can lead to system-level performance degradation. Specifically, when one or more microphones in the array experience abnormal sound reception due to physical damage, blocked microphone holes, loose connections, circuit failures, or electromagnetic interference, their output audio signals will contain distortion or noise components. This abnormal signal is used in sound source localization, beamforming, and noise reduction processes, causing signal processing algorithms to fail to accurately extract valid voice information. This results in a decrease in voice wake-up rate, an increase in false wake-up rate, and a reduction in voice recognition accuracy, ultimately affecting the device's reliability in responding to voice commands. For example, in a smart TV box deployed in a home living room, one microphone's microphone hole may become blocked by fine dust particles due to prolonged exposure to air. When a user issues a voice command, the audio signal energy characteristics output by this microphone deviate significantly from the normal range, and its cross-correlation characteristics with other microphones show abnormal deviations. Consequently, the system cannot effectively focus on the direction of the sound source when performing beamforming operations, causing voice commands to be incorrectly identified as environmental noise or completely missed. This manifests as the device not responding to valid commands or falsely triggering non-command sounds.

[0027] If the above problems are not resolved, the abnormal signals from the faulty microphones will continue to contaminate the overall data stream of the microphone array, introducing irreversible errors into the sound field reconstruction process. These errors will further propagate to subsequent voice processing stages, weakening the device's adaptability to complex acoustic environments, gradually eroding the stability and usability of voice interaction functions, and ultimately preventing users from operating the device normally via voice, severely damaging the product's market reputation and user satisfaction.

[0028] Based on this, embodiments of this application provide a microphone anomaly handling method, apparatus, device, and medium. By evaluating features to identify abnormal microphones, and reconstructing signals based on audio sampling signals from non-abnormal microphones and the location of the microphones, adaptive processing can be performed on microphone anomalies in a microphone array, ensuring that voice interaction devices operate normally when microphones malfunction, and improving the voice interaction experience.

[0029] Figure 1 This diagram illustrates the application environment of the microphone anomaly handling method provided in this embodiment. (See also...) Figure 1 This method is applied to a microphone anomaly handling system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of several servers. The terminal 110 sends audio sampling signals obtained from multiple microphones in the microphone array to the server 120. The server 120 acquires the audio sampling signals obtained from multiple microphones in the microphone array, evaluates the target features of the audio sampling signals, obtains feature evaluation results, identifies abnormal microphones with signal anomalies based on the feature evaluation results, reconstructs the audio signals at the location of the abnormal microphones based on the audio sampling signals of the non-abnormal microphones and the microphones' locations, obtains reconstructed audio signals, and inputs the audio sampling signals of the non-abnormal microphones and the reconstructed audio signals into a voice interaction device to enable the voice interaction device to perform corresponding voice interaction operations. The target features include the signal power characteristics, signal energy characteristics, signal-to-noise ratio characteristics, cross-correlation characteristics, coherence characteristics, and / or spectral characteristics of the audio sampling signals.

[0030] It should be understood that Figure 1The application scenarios shown are merely examples. In practical applications, the microphone anomaly handling method provided in this application embodiment can also be applied to other scenarios. For example, the above-described microphone anomaly handling method can be directly applied to terminal 110. Terminal 110 is used to acquire audio sampling signals obtained from multiple microphones in a microphone array, evaluate the target features of the audio sampling signals, obtain feature evaluation results, determine abnormal microphones with signal anomalies based on the feature evaluation results, and reconstruct the audio signals at the location of the abnormal microphones based on the audio sampling signals of the non-abnormal microphones and the location of the microphones, obtain audio reconstruction signals, and input the audio sampling signals of the non-abnormal microphones and the audio reconstruction signals into a voice interaction device so that the voice interaction device can perform corresponding voice interaction operations.

[0031] See Figure 2 In one embodiment, a microphone anomaly handling method is provided. The execution subject of the method can be a terminal or a server, including but not limited to steps S201 to S205.

[0032] Step S201: Obtain audio sampling signals obtained from multiple microphones in the microphone array.

[0033] A microphone array is a system consisting of multiple microphones arranged in a specific geometric pattern. It is used to collect sound field information in a coordinated manner to achieve functions such as sound source localization, beamforming, and noise reduction.

[0034] Audio sampling signals refer to discrete digital signals obtained by analog-to-digital conversion after a microphone converts sound waves into electrical signals, representing acoustic information within a specific time period.

[0035] Each microphone in the microphone array is configured to independently acquire acoustic information from its location and convert it into a digital audio sample signal. These signals are then transmitted to the microphone processing unit. For example, each microphone in a four-microphone array continuously acquires ambient sound at a preset sampling rate and bit depth, generating a corresponding digital audio sample signal, which is then aggregated by the microphone processing unit and acquired by the execution unit.

[0036] Step S202: Evaluate the target features of the audio sampling signal to obtain the feature evaluation results.

[0037] Target characteristics include signal power characteristics, signal energy characteristics, signal-to-noise ratio characteristics, cross-correlation characteristics, coherence characteristics, and / or spectral characteristics of the audio sampled signal. In essence, target characteristics refer to specific signal attribute features used to evaluate the quality or state of the audio sampled signal; these features reflect the microphone's operating state and signal integrity.

[0038] Signal power characteristics refer to the average power of an audio sampled signal over a specific time period, reflecting the signal's intensity. Signal energy characteristics refer to the total energy of an audio sampled signal over a specific time period; similar to signal power, it also reflects the signal's intensity. Signal-to-noise ratio (SNR) characteristics refer to the ratio of the power of the effective signal to the power of the noise signal in an audio sampled signal, reflecting its clarity. Cross-correlation characteristics are a measure of the similarity between two or more audio sampled signals, used to assess the consistency of audio sampled signals from different microphones in a microphone array. Coherence characteristics refer to the correlation between two or more audio sampled signals in the frequency domain, reflecting the phase and amplitude relationships between different microphones. Spectral characteristics refer to the distribution characteristics of an audio sampled signal in the frequency domain, such as the intensity or trend of specific frequency components.

[0039] The target characteristics of the audio sampling signal can be evaluated by a human operator who listens to the audio output of each microphone and visually inspects the waveform to determine the signal quality of each microphone and records the evaluation results. In another implementation, a series of simple rules can be set. For example, a lower limit for signal power can be preset; when the signal power of an audio sampling signal from a microphone is below this lower limit, it is initially marked as potentially abnormal. Alternatively, an upper limit for signal-to-noise ratio can be preset; when the signal-to-noise ratio of an audio sampling signal from a microphone is above this upper limit, it is also initially marked as potentially abnormal. Furthermore, statistical analysis can be performed on various target characteristics of the audio sampling signal from each microphone, such as calculating the average signal energy over a period of time, and preliminary classification judgments can be made based on these statistical values.

[0040] Step S203: Based on the feature evaluation results, identify abnormal microphones with signal anomalies.

[0041] Based on the results of the aforementioned feature evaluation, the executing entity can identify which microphones exhibit abnormal signal characteristics. For example, if a human operator's evaluation indicates that a microphone's reception is distorted or excessively noisy, that microphone is identified as abnormal. Alternatively, if a microphone's signal power consistently falls below a preset lower limit, that microphone is automatically identified as abnormal. Furthermore, if statistical analysis reveals a difference between the spectral characteristics of a microphone and those of other microphones in the array, that microphone is also identified as abnormal.

[0042] Step S204: Based on the audio sampling signal of the non-abnormal microphone and the location of the microphone, the audio signal at the location of the abnormal microphone is reconstructed to obtain the audio reconstructed signal.

[0043] Signal reconstruction refers to the process of using audio sampling signals and location information from non-abnormal microphones to estimate and generate the expected audio signal at the location of the abnormal microphone through algorithms or models.

[0044] Audio reconstruction signal refers to a synthesized audio signal obtained through a signal reconstruction process and used to replace the original signal from a faulty microphone.

[0045] Reconstructing the audio signal at the location of the abnormal microphone can be achieved by directly copying the audio sample signal from the nearest non-abnormal microphone and using it as the reconstructed signal. Alternatively, the expected audio signal at the location of the abnormal microphone can be estimated and generated using a linear interpolation algorithm, utilizing the audio sample signals and spatial location information of two or three non-abnormal microphones surrounding the abnormal microphone. Furthermore, the audio sample signals from all non-abnormal microphones can be averaged, and the averaged signal can be used as the reconstructed signal for the abnormal microphone's location.

[0046] Step S205: Input the audio sampling signal and audio reconstruction signal of the non-abnormal microphone into the voice interaction device so that the voice interaction device can perform the corresponding voice interaction operation.

[0047] Voice interaction devices are devices that can receive and process audio sampling signals and audio reconstruction signals, and perform corresponding operations according to voice commands, such as smart speakers, smart TVs, or smart terminals.

[0048] Before inputting the audio sampling signal and the reconstructed audio signal from the non-abnormal microphone into the voice interaction device, the original audio sampling signals from all non-abnormal microphones and the reconstructed audio signal from the abnormal microphone can be integrated into a complete microphone array output and transmitted to the voice interaction device. Upon receiving these integrated signals, the voice interaction device treats them as a fully functional, fault-free microphone array input, thus enabling it to perform subsequent voice interaction operations such as sound source localization, beamforming, speech recognition, and voice wake-up.

[0049] The following example will provide a more detailed explanation of the above technical solution: Suppose a smart speaker with a microphone array of four microphones is deployed at location A in a smart home environment, serving as a voice interaction device. Over time, one of the microphones, such as microphone 1, may experience abnormal sound pickup due to dust clogging its microphone hole, resulting in decreased signal power and a deteriorated signal-to-noise ratio.

[0050] First, the smart speaker's microphone array continuously acquires audio sample signals from four microphones. These signals are then transmitted in real time to the processing module inside the smart speaker.

[0051] Next, the processing module performs feature evaluation on the target features of the audio sampling signals from each microphone. Specifically, the processing module calculates the signal power characteristics and signal-to-noise ratio characteristics of each microphone over a recent period. For example, it calculates the average signal power of microphones 1, 2, 3, and 4.

[0052] Subsequently, based on the feature evaluation results, the processing module identifies abnormal microphones with signal anomalies. Assume the processing module finds that the average signal power of microphone 1 is significantly lower than that of microphones 2, 3, and 4, and that microphone 1's signal-to-noise ratio is also lower than the other microphones. Therefore, the processing module identifies microphone 1 as an abnormal microphone, while microphones 2, 3, and 4 are identified as normal microphones.

[0053] Furthermore, based on the audio sampling signals of the non-abnormal microphones (microphone 2, microphone 3, and microphone 4) and their locations, the processing module reconstructs the audio signal at the location of the abnormal microphone (microphone 1) to obtain an audio reconstruction signal. For example, the processing module can use the audio sampling signals of microphones 2, 3, and 4, combined with their geometric positions in the microphone array, to estimate the expected sound field information at the location of microphone 1 using a spatial interpolation algorithm, and generate an audio reconstruction signal simulating the normal operation of microphone 1.

[0054] Finally, the processing module takes the original audio sampling signals from microphones 2, 3, and 4, as well as the reconstructed audio signal from microphone 1, and provides them as a complete, "repaired" microphone array input to the smart speaker's voice recognition engine. This allows the smart speaker's voice recognition engine to receive a high-quality, complete array signal, enabling it to perform voice interaction operations such as voice wake-up and voice command recognition. For example, if user A issues the wake-up command "Hello, smart speaker," the smart speaker can still accurately recognize and respond even if microphone 1 malfunctions, without failing to wake up or making recognition errors due to microphone 1's abnormality.

[0055] In some embodiments, when a user's TV box has a microphone hole blocked by dust, the execution unit automatically identifies the anomaly and uses the remaining microphones to calculate the virtual audio at that location in real time. When the user wakes up Google Assistant, a far-field wake-up success rate of over 5 meters is still maintained, and no factory repair is required.

[0056] Based on the above examples, the technical solution provided in this embodiment demonstrates a technological contribution. In the prior art, when a microphone in a microphone array malfunctions, such as microphone 1 being blocked, the abnormal or low-quality signals it collects will be directly input to the voice interaction device, resulting in a decrease in the performance of the entire microphone array. This typically manifests as a decrease in voice wake-up rate, a decrease in voice recognition accuracy, and may even cause the device to be unable to respond to the user's voice commands, thereby impairing the user experience and the reliability of the device.

[0057] In contrast, this embodiment solves the aforementioned technical problems by introducing feature evaluation and signal reconstruction mechanisms. Specifically, by evaluating the target features of the audio sampling signals from each microphone in the microphone array, abnormal microphones can be identified promptly and accurately. For example, in the example above, abnormal signal power and signal-to-noise ratio of microphone 1 are detected. This step avoids directly transmitting abnormal data from faulty microphones to subsequent processing stages.

[0058] Furthermore, this embodiment utilizes the audio sampling signals and location information of the non-malfunctioning microphones to reconstruct the audio signal at the location of the malfunctioning microphone. In the example above, the normal signals from microphones 2, 3, and 4 are used to generate the reconstructed audio signal for microphone 1. This reconstruction process compensates for the functional deficiencies of the malfunctioning microphone, enabling the voice interaction device to receive a complete and high-quality microphone array input. This processing method improves the robustness of the microphone array, ensuring the overall system maintains stable performance even if some microphones fail.

[0059] Finally, the original signal and reconstructed signal from the non-malfunctioning microphone are input into the voice interaction device, ensuring that the device can continuously perform corresponding voice interaction operations. In the example above, user A's wake-up command can be accurately recognized by the smart speaker, even if microphone 1 is in an abnormal state. Therefore, the technical solution of this embodiment can avoid damage to the voice interaction function due to microphone failure, thereby maintaining the user experience and improving the reliability of the device.

[0060] See Figure 3 In one embodiment, the method for evaluating the target features of the audio sampling signal includes, but is not limited to, steps S301 to S304.

[0061] Step S301: Determine the feature deviation between the target feature and the reference feature of the audio sampling signal.

[0062] The reference feature is either the average value of the target features from other audio sampled signals or a preset target feature threshold. It can be understood that the reference feature is a benchmark used for comparison with the target features of the microphone being evaluated. This benchmark can be dynamically generated, for example, by calculating the average value of the same target features from the audio sampled signals of other non-abnormal microphones in the microphone array, allowing the reference feature to adapt to environmental changes. Alternatively, the benchmark can be a pre-set fixed value, i.e., a preset target feature threshold, which is typically determined based on empirical data or system design requirements to define the boundaries of the normal operating range.

[0063] The feature deviation between the target feature and the reference feature of an audio sampling signal refers to the difference between the feature value of the target feature and the feature value of a certain reference feature in the audio sampling signal of the microphone being evaluated. This difference can be an absolute value, used to measure the magnitude of the deviation, or a relative value, used to measure the proportion of the deviation. For example, the feature deviation can be expressed as the difference between the feature value of the target feature and the feature value of the reference feature, or the ratio of the difference to the feature value of the reference feature.

[0064] Step S302: Determine whether the feature deviation value meets the preset feature deviation conditions.

[0065] If it does not meet the requirements, proceed to step S303; if it does meet the requirements, proceed to step S304.

[0066] Feature deviation criteria are rules or standards used to determine whether a microphone's state is abnormal. This criterion is typically a logical expression; for example, if the absolute value of the feature deviation exceeds a preset threshold, or if the feature deviation falls outside a specific range, then the criterion is considered not met.

[0067] Step S303: Generate feature evaluation results for microphone anomalies.

[0068] Step S304: Generate feature evaluation results for microphone non-abnormalities.

[0069] This application provides a quantitative and systematic implementation of the feature evaluation step in microphone anomaly handling methods. First, for the audio sampling signal of each microphone in the microphone array, target features, such as signal power features, are extracted. Then, this target feature is compared with a reference feature, which can be the average signal power of other microphones in the array or a preset normal signal power threshold. By calculating the feature deviation between the two, the degree of deviation between the current microphone's operating state and its normal state can be quantitatively reflected. Next, this feature deviation is compared with preset feature deviation conditions. For example, if the feature deviation (such as the decrease in signal power features) exceeds a preset tolerance range, the microphone is judged to be abnormal. This evaluation mechanism based on quantitative comparison and conditional judgment makes the identification process of abnormal microphones more objective, accurate, and automated, avoiding misjudgments that may be caused by relying solely on experience or simple observation, thus providing more reliable input for subsequent signal reconstruction and voice interaction operations.

[0070] The following example illustrates this. Suppose a microphone array contains four microphones. When handling microphone anomalies, the audio sample signal of each microphone is first acquired. For each microphone, its signal power is calculated as a target feature. To assess whether the signal power of microphone A is abnormal, reference features can be determined in two ways: One approach is to calculate the average signal power of microphones B, C, and D as a reference feature. Then, the difference between the signal power of microphone A and this average value is calculated as the feature deviation. A preset feature deviation condition could be: if the absolute value of the feature deviation is greater than 10 dB, it is considered to fail the condition. If the signal power of microphone A is 15 dB lower than the average value of the other microphones, the feature deviation fails the condition, thus generating a feature evaluation result indicating an abnormality for microphone A.

[0071] Another approach is to preset a signal power threshold, such as -50 dBFS, as a reference feature. Then, the difference between the signal power of microphone A and -50 dBFS is calculated as the feature deviation. The preset feature deviation condition could be: if the feature deviation causes the signal power to fall below -60 dBFS, it is considered to fail the condition. If the signal power of microphone A is -70 dBFS, the feature deviation fails the condition, thus generating a feature evaluation result indicating an abnormality in microphone A.

[0072] By using any of the above methods, abnormal microphones can be accurately identified, providing accurate abnormal microphone information for subsequent signal reconstruction.

[0073] Through the above technical solution, this application provides a more accurate and automated microphone anomaly assessment mechanism. By quantitatively comparing the deviation between the target features and reference features of the audio sampling signal and making judgments based on preset conditions, the limitations of subjective judgment or simple threshold determination can be effectively avoided, significantly improving the accuracy and reliability of abnormal microphone identification. This not only reduces the risk of false alarms and missed alarms but also provides more accurate abnormal microphone information for subsequent signal reconstruction steps, thereby ensuring that the voice interaction device can still stably and efficiently perform voice interaction operations when the microphone is partially or completely abnormal, improving the robustness of the system and the user experience.

[0074] See Figure 4 In one embodiment, the method for reconstructing the audio signal at the location of the abnormal microphone includes, but is not limited to, steps S401 to S403.

[0075] Step S401: Based on the location of the non-abnormal microphones, reconstruct the effective geometric topology of the microphone array.

[0076] Step S402: Based on the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone, determine the signal reconstruction model corresponding to the location of the abnormal microphone.

[0077] The signal reconstruction model is used to characterize the mapping relationship between the audio sampled signal of the non-abnormal microphone and the expected audio signal at the location of the abnormal microphone.

[0078] Step S403: Based on the signal reconstruction model, process the audio sampling signal of the non-abnormal microphone to generate an audio reconstruction signal corresponding to the location of the abnormal microphone.

[0079] Reconstructing the effective geometric topology of a microphone array based on the location of non-abnormal microphones can be achieved by building a geometric structure that accurately reflects the relative spatial relationships of the non-abnormal microphones, based on the spatial position information of the microphones currently operating normally in the array. For example, the complete geometric layout of the microphone array can be pre-stored. When non-abnormal microphones are identified, the execution entity can extract the precise spatial coordinates of these non-abnormal microphones from the complete layout, thereby forming an effective geometric topology. Alternatively, the real-time spatial positions of non-abnormal microphones can be dynamically measured and determined using sound source localization or array calibration techniques during the execution entity's startup or runtime, thereby constructing the currently usable effective geometric topology.

[0080] After reconstructing the effective geometric topology of the microphone array, a signal reconstruction model corresponding to the location of the abnormal microphone is determined. This signal reconstruction model can infer the expected audio signal that the abnormal microphone should have collected based on the audio sampling signal collected by the non-abnormal microphone, the known effective geometric topology, and the precise spatial location of the abnormal microphone. The core of this signal reconstruction model lies in capturing the mapping relationship between the non-abnormal microphone signal and the sound field at the location of the abnormal microphone. For example, this model can be a model based on spatial interpolation algorithms, such as using inverse distance weighted (IDW) interpolation, radial basis function (RBF) interpolation, or Kriging interpolation, to estimate the sound field value at the location of the abnormal microphone based on the spatial distribution and signal strength of the non-abnormal microphones. Alternatively, this model can also be a machine learning model, such as a pre-trained neural network model, which learns from a large amount of normal and abnormal microphone array data to establish a complex nonlinear mapping from input (non-abnormal microphone signal, geometric topology, abnormal location) to output (expected signal from the abnormal microphone).

[0081] By calculating or inferring using the established signal reconstruction model, an estimated signal at the location of the abnormal microphone is obtained, i.e., an audio reconstruction signal corresponding to the location of the abnormal microphone is generated. This generated audio reconstruction signal will be used to replace the missing or damaged signal of the abnormal microphone. For example, if the signal reconstruction model is based on spatial interpolation, the audio sampling signal of the non-abnormal microphone is used as the input data point of the interpolation algorithm, and the signal value at the location of the abnormal microphone is calculated by combining its spatial location with the location of the abnormal microphone. If the signal reconstruction model is a pre-trained neural network, the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone are used as the input features of the neural network, and the audio reconstruction signal output by the network is obtained through forward propagation.

[0082] This application's solution first reconstructs the effective geometric topology of the microphone array based on the location of the non-abnormal microphones, clarifying the currently available microphone spatial layout and providing a precise spatial reference for subsequent signal reconstruction. Based on this, a signal reconstruction model is systematically determined by combining the audio sampling signals from the non-abnormal microphones, the determined effective geometric topology, and the location of the abnormal microphone. This model is specifically designed to characterize the mapping relationship between the non-abnormal microphone signals and the expected signals at the location of the abnormal microphone, thus transforming the complex sound field reconstruction problem into a computable model problem. Finally, by inputting the audio sampling signals from the non-abnormal microphones into this signal reconstruction model for processing, an audio reconstruction signal corresponding to the location of the abnormal microphone can be generated. This step-by-step and structured approach ensures the accuracy and effectiveness of the signal reconstruction process, avoids blind or empirical signal filling, significantly improves the quality of abnormal microphone signal recovery, and thus guarantees the normal operation of the voice interaction device in the event of partial microphone failure.

[0083] As a specific implementation, suppose a microphone array contains four microphones, numbered M1, M2, M3, and M4, arranged linearly with equal spacing. In a single detection, microphone M1 is identified as an abnormal microphone, while M2, M3, and M4 are non-abnormal microphones. The execution unit first reconstructs the effective geometric topology of the microphone array based on the locations of M2, M3, and M4. For example, if the original array's geometric topology is M1(0,0,0), M2(0.1,0,0), M3(0.2,0,0), and M4(0.3,0,0), then the effective geometric topology will only include the location information of the three non-abnormal microphones: M2(0.1,0,0), M3(0.2,0,0), and M4(0.3,0,0). Subsequently, based on the audio sampling signals of M2, M3, and M4, the aforementioned effective geometric topology, and the location of the abnormal microphone (0,0,0), a signal reconstruction model is determined. The model can be an adaptive filter based on the Least Mean Square Error (LMS) algorithm, used to learn the linear or nonlinear relationship between the signals M2, M3, and M4 and the expected signal M1. Once the model is determined, the audio sampled signals of M2, M3, and M4 can be input into the model in real time for processing, thereby generating an audio reconstruction signal corresponding to the location of M1. The reconstructed M1' signal must be strictly synchronized with the signals M2, M3, and M4 in terms of timestamp and phase. For example, the model can infer the expected signal of M1 in the current sound field based on the signals of M2, M3, and M4 through weighted averaging or more complex filtering operations.

[0084] Through the above technical solution, this application provides a structured and efficient signal reconstruction mechanism. By clearly defining the effective geometric topology of the reconstructed microphone array, it provides a precise spatial basis for signal reconstruction, avoiding blind processing under incomplete information. Furthermore, by determining and applying a specialized signal reconstruction model, it can accurately capture the mapping relationship between non-abnormal microphone signals and expected signals at abnormal microphone locations, thereby generating high-quality audio reconstruction signals. This significantly improves the accuracy and reliability of abnormal microphone signal recovery, effectively solving the technical problem of how to efficiently and accurately utilize non-abnormal microphone information for signal reconstruction when the microphone is partially abnormal. It ensures that the voice interaction device can still provide stable and accurate voice input when facing microphone abnormalities, guaranteeing the user experience and the integrity of device functionality.

[0085] In some embodiments, the signal reconstruction model is a spatial sound field interpolation-based model. Based on the signal reconstruction model, the audio sampling signals of non-abnormal microphones are processed, including: calculating the spatial correlation characteristics between non-abnormal microphones based on the audio sampling signals of non-abnormal microphones and the effective geometric topology; estimating the sound field characteristics at the location of the abnormal microphone based on the spatial correlation characteristics and the location of the abnormal microphone using the signal reconstruction model; and generating an audio reconstruction signal based on the sound field characteristics at the location of the abnormal microphone.

[0086] A spatial sound field interpolation-based model is a mathematical model that uses acoustic measurement data from known locations to estimate the sound field characteristics at unknown locations. The core of this model lies in utilizing the propagation laws of sound waves in space and the correlation between sound fields at adjacent locations to infer the sound field information at the location of an abnormal microphone through interpolation algorithms. This model can effectively capture the continuity and changing trends of the sound field in space, providing a foundation for accurate audio signal reconstruction. For example, this model can be implemented using methods such as Kriging interpolation, inverse distance weighting (IDW) interpolation, or radial basis function (RBF) interpolation.

[0087] Calculating the spatial correlation characteristics between non-abnormal microphones refers to quantifying the degree of spatial correlation between audio signals collected by different non-abnormal microphones in a microphone array. This characteristic reflects the influence of the distance and angle between the sound source and different microphones, as well as the sound field environment (such as reflections and reverberation) on the signal. By calculating spatial correlation characteristics, the similarity or dependence between different microphone signals can be quantified, which is crucial for accurate spatial sound field interpolation. For example, the cross-correlation coefficient, coherence function, or spatial covariance matrix can be calculated between the audio sampling signals of non-abnormal microphones to characterize their spatial correlation characteristics.

[0088] Estimating the acoustic field characteristics at the location of an abnormal microphone involves, after determining the spatial correlation characteristics between non-abnormal microphones, using this known information based on a spatial acoustic field interpolation model to predict the acoustic field properties at the location of the abnormal microphone, such as sound pressure, sound intensity, and direction. This process is a crucial step in signal reconstruction, transforming the abstract acoustic field model into concrete acoustic parameter estimates. For example, the instantaneous sound pressure level or its spectral characteristics at the location of the abnormal microphone can be estimated.

[0089] This application's solution first utilizes the audio sampling signals from non-abnormal microphones and the effective geometric topology of the microphone array to calculate the spatial correlation characteristics among these microphones. These spatial correlation characteristics reveal the intrinsic relationship between the sound field at different microphone locations, providing a data foundation for subsequent interpolation. Subsequently, combining the calculated spatial correlation characteristics with the precise location of the abnormal microphone, a precise estimation of the sound field characteristics at the location of the abnormal microphone is performed based on a pre-defined spatial sound field interpolation model. This estimation process fully utilizes the spatial distribution information of the non-abnormal microphones and the interrelationships between their signals, thereby enabling a more accurate prediction of the acoustic environment at the location of the abnormal microphone. Finally, based on the estimated sound field characteristics at the location of the abnormal microphone, a corresponding audio reconstruction signal is generated. In this way, this solution overcomes the problem of insufficient utilization of spatial information that may exist in traditional signal reconstruction methods, ensuring that the reconstructed audio signal is not only similar to the original signal in the time domain, but also highly consistent with the actual situation in terms of spatial sound field characteristics, thus providing high-quality input for voice interaction devices.

[0090] The following is a concrete example. When a microphone in a microphone array is identified as an anomalous microphone, the executing entity can first obtain the audio sampling signals of all non-abnormal microphones and the effective geometric topology of the microphone array. For example, a cross-correlation function can be used to calculate the spatial correlation characteristics between the audio sampling signals of any two non-abnormal microphones, resulting in a spatial correlation matrix. Assume the signal reconstruction model uses a Kriging interpolation model, which can predict the values ​​of unknown points based on the measurements of known points and their spatial correlations. In this case, the calculated spatial correlation characteristics, the audio sampling signals of the non-abnormal microphones, and the precise spatial coordinates of the anomalous microphone are input into the Kriging interpolation model. The Kriging interpolation model uses this information to estimate the sound pressure characteristics at the location of the anomalous microphone by establishing an optimal linear unbiased estimator. Once the sound pressure characteristics at the location of the anomalous microphone (e.g., a series of sound pressure values ​​varying over time) are obtained, the executing entity can use digital signal processing techniques, such as converting the sound pressure values ​​into digital audio samples, to generate an audio reconstruction signal corresponding to the location of the anomalous microphone.

[0091] The above technical solution specifically defines the signal reconstruction model as a spatial sound field interpolation model. It clarifies that the sound field characteristics at the location of the abnormal microphone are estimated by calculating the spatial correlation characteristics between non-abnormal microphones, thereby generating the audio reconstruction signal. This allows the reconstruction process to fully utilize the spatial information of the microphone array and the continuity of the sound field in space, significantly improving the accuracy and realism of the audio reconstruction. Compared to reconstruction methods that rely solely on general mapping relationships, this solution can more accurately capture the acoustic environment of the abnormal microphone location, effectively compensating for signal loss caused by microphone abnormalities. This provides voice interaction devices with audio input closer to real-world scenarios, greatly improving the reliability of voice interaction and user experience.

[0092] In some embodiments, the signal reconstruction model is a pre-trained neural network model. Based on the signal reconstruction model, the audio sampling signals of non-abnormal microphones are processed, including: inputting the audio sampling signals of non-abnormal microphones, the effective geometric topology, and the location of abnormal microphones into the signal reconstruction model; and obtaining the audio reconstruction signal output by the signal reconstruction model. The signal reconstruction model is trained based on the audio sample signals, sample geometric topology, and sample abnormal locations.

[0093] The signal reconstruction model can be a deep learning model, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer network, or a variant thereof. These models can effectively handle temporal data and spatial correlations. During signal reconstruction, the audio sampling signals from the non-abnormal microphones, the effective geometric topology, and the location of the abnormal microphone are input into the signal reconstruction model. These inputs provide the model with all the contextual information needed for signal reconstruction, enabling it to comprehensively consider sound source information, the physical layout of the microphone array, and the target reconstruction location, thereby accurately estimating the audio signal at the location of the abnormal microphone. For example, the audio sampling signals can be input as time-domain waveforms, frequency-domain spectra (such as short-time Fourier transform STFT), or Mel-frequency cepstral coefficients (MFCC); the effective geometric topology can be represented as a vector or matrix of microphone coordinates; and the location of the abnormal microphone can be represented as three-dimensional coordinates.

[0094] It's important to note that the signal reconstruction model establishes its internal parameters by learning on a large amount of sample data. The audio sample signals are real or simulated audio data used for training, covering various acoustic environments, sound source types, and noise conditions. The sample geometry and topology are the microphone array configuration information corresponding to the audio sample signals, including microphone arrays of different numbers and layouts. Sample anomaly locations indicate the simulated locations of anomalous microphones during training. In this way, the neural network model can learn the complex nonlinear mapping relationship between non-anomalous microphone signals and the target reconstructed signal under different acoustic scenarios, microphone array configurations, and anomaly locations. For example, the training process can employ a supervised learning paradigm, where the model's input is the simulated non-anomalous microphone signal, geometry and topology, and anomaly location, while the model's output is compared with the real (or simulated) audio signal at that anomaly location, and the model parameters are adjusted by optimizing the loss function.

[0095] After the signal reconstruction model receives and processes the input data, its output is the required audio reconstruction signal. This output signal is the optimal estimate of the audio signal at the location of the faulty microphone, based on the knowledge learned during the training phase. In this way, the functional deficiencies of the faulty microphone can be effectively compensated for, providing complete audio information for subsequent voice interaction operations. For example, the audio reconstruction signal output by the model can be time-domain waveform data or frequency-domain representation, which can be converted back to time-domain waveform for use by the voice interaction device.

[0096] This application's solution significantly improves the robustness and accuracy of microphone anomaly handling by introducing a pre-trained neural network model as the signal reconstruction model. After determining the presence of an abnormal microphone, the system first reconstructs the effective geometric topology of the microphone array based on the locations of the non-abnormal microphones. Then, the audio sampling signal from the current non-abnormal microphone, the reconstructed effective geometric topology, and the location of the abnormal microphone are used as inputs to the pre-trained neural network model. During the training phase, this neural network model has learned and internalized the complex mapping relationship between the non-abnormal microphone signal and the expected audio signal at the location of the abnormal microphone through a large number of audio sample signals, sample geometric topologies, and sample anomaly locations. Therefore, when receiving real-time input, the model can intelligently process the audio sampling signal from the non-abnormal microphone using its learned knowledge, thereby accurately estimating and generating the audio reconstruction signal at the location of the abnormal microphone. This data-driven, deep learning-based reconstruction method can effectively cope with complex acoustic environments, varied microphone array configurations, and various types of microphone anomalies that are difficult for traditional models to handle. This overcomes the limitations of traditional signal reconstruction methods in terms of model design and adaptability, ensuring that even in the case of partial microphone anomalies, voice interaction devices can still obtain high-quality audio input and thus perform accurate voice interaction operations.

[0097] In some embodiments, before determining the signal reconstruction model corresponding to the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone, the method further includes: estimating the device performance characteristics of the non-abnormal microphone based on the feature evaluation results; and performing signal compensation on the audio sampling signal of the non-abnormal microphone according to the device performance characteristics of the non-abnormal microphone to obtain the compensated audio sampling signal of the non-abnormal microphone.

[0098] The proposed solution significantly improves the accuracy and reliability of signal reconstruction by introducing estimation and signal compensation steps for the device performance characteristics of non-abnormal microphones before determining the signal reconstruction model. Specifically, in a microphone array, the non-abnormal microphones are first identified through feature evaluation. Subsequently, based on these feature evaluation results, the device performance characteristics of each non-abnormal microphone are further estimated, thereby quantifying its specific performance in audio capture. For example, it can be identified which non-abnormal microphones have low sensitivity, which have frequency response deviations, or which have high inherent noise levels. Once these device performance characteristics are obtained, the system performs targeted signal compensation on the raw audio sampling signals of each non-abnormal microphone based on these characteristics. This compensation operation aims to eliminate or reduce signal distortion or inconsistency caused by individual microphone performance differences, making the audio sampling signals output by all non-abnormal microphones more standardized and consistent in quality and characteristics. For example, gain compensation is performed on microphones with low sensitivity, equalization compensation is performed on microphones with uneven frequency responses, or noise reduction processing is performed on microphones with high noise levels. After compensation processing, the quality and consistency of the compensated audio sampling signals of the non-abnormal microphones are significantly improved. These high-quality, consistent signals were then used to reconstruct the effective geometric topology of the microphone array. Based on this, and combined with the location of the abnormal microphones, a more accurate signal reconstruction model was determined. Due to the higher quality of the input signal, this signal reconstruction model can more accurately represent the mapping relationship between the audio sample signals of the non-abnormal microphones and the expected audio signals at the location of the abnormal microphones. Finally, based on this optimized signal reconstruction model, the compensated audio sample signals of the non-abnormal microphones are processed to generate more accurate, high-quality audio reconstruction signals, thereby providing more reliable input for voice interaction devices.

[0099] The following is a concrete example to illustrate this. Assume a microphone array contains multiple microphones. After initial feature evaluation, some microphones are identified as non-abnormal. To further optimize signal reconstruction, the system estimates the device performance characteristics of each non-abnormal microphone based on its feature evaluation results, such as historical signal-to-noise ratio data and frequency response test results. For example, microphone A is estimated to have slightly below-average sensitivity, while microphone B is estimated to have slight attenuation in a specific high-frequency range. Based on this, the system performs signal compensation on the audio sampling signals of these non-abnormal microphones. Specifically, for the audio sampling signal of microphone A, a preset gain factor can be applied to amplify it to compensate for its insufficient sensitivity. For the audio sampling signal of microphone B, a digital equalizer filter can be applied to compensate for its high-frequency attenuation, making its frequency response flatter. After such compensation, the audio sampling signals of microphones A and B become more consistent and optimized in quality and characteristics. These compensated audio sampling signals are then input into the signal reconstruction module to build a more accurate signal reconstruction model and ultimately generate a high-quality audio reconstruction signal.

[0100] By employing the aforementioned technical solution, before determining the signal reconstruction model, the device performance characteristics of non-abnormal microphones are estimated and their audio sampling signals are compensated, effectively solving the problem of decreased signal reconstruction accuracy caused by the performance differences of non-abnormal microphones. Standardizing and optimizing the audio sampling signals of non-abnormal microphones ensures higher consistency and quality of the input signals used for signal reconstruction. This allows the subsequently determined signal reconstruction model to more accurately capture sound field information, thereby generating audio reconstruction signals that are closer to reality. Ultimately, this improves the accuracy and reliability of audio signal reconstruction at abnormal microphone locations, enhances the robustness of the entire microphone array system, and provides higher-quality audio input for voice interaction devices to perform corresponding voice interaction operations.

[0101] In some embodiments, after determining that an abnormal microphone exists, the method further includes one or more of the following steps: generating an abnormal prompt message, shortening the pickup distance of the non-abnormal microphones, and switching the voice interaction device to a near-field interaction mode according to the number of abnormal microphones; switching to noise reduction operation on the audio sampling signal of a single non-abnormal microphone; and restricting the voice interaction device from performing a preset target voice interaction operation.

[0102] Generating anomaly alert messages refers to the warning notification sent to the user or relevant management system after an abnormal microphone is detected in the microphone array. This information aims to promptly inform the user of the microphone array's operational status so that the user is aware of the current device malfunction and the potential need for intervention. Anomaly alert messages can be implemented in various ways. For example, they can be displayed as text on the voice interaction device's screen saying "Microphone array malfunction, please check," or announced via voice broadcast as "Microphone malfunction detected; some functions may be restricted." Notifications can also be sent to associated mobile applications or remote management platforms via the network.

[0103] Shortening the pickup distance of non-malfunctioning microphones refers to adjusting their pickup parameters to primarily capture nearby sound sources when a malfunctioning microphone is present in the microphone array. The aim is to effectively suppress interference from distant noise and reverberation by focusing on near-field speech when part of the microphone array fails, thereby improving the clarity and recognition rate of near-field speech and compensating for the performance loss caused by the malfunctioning microphone. This can be achieved by adjusting the gain settings of the non-malfunctioning microphones to reduce their response to far-field sounds, or by adjusting the parameters of the beamforming algorithm to narrow its main lobe width and more accurately target nearby sound sources.

[0104] Switching to noise reduction based on the audio sample signal from a single, non-abnormal microphone means that when an abnormal microphone is present in the microphone array, the system no longer uses complex array noise reduction algorithms that typically rely on the spatial information of multiple microphones. Instead, it selects one or more high-performance, non-abnormal microphones and applies a noise reduction algorithm suitable for a single microphone to their audio sample signals. This switching aims to avoid the negative impact of erroneous information introduced by abnormal microphones on the overall noise reduction effect, ensuring that at least one microphone can provide an effectively noise-reduced speech signal. Specific noise reduction algorithms can employ single-channel noise reduction techniques based on spectral subtraction, Wiener filtering, or deep learning.

[0105] Restricting voice interaction devices from performing preset target voice interaction operations refers to the system proactively disabling or limiting the execution of certain preset functions or commands by the voice interaction device when the microphone array performance is impaired. This is done to avoid misrecognition or erroneous operations caused by decreased voice recognition accuracy, thereby protecting the user experience and improving system security. For example, sensitive operations requiring high voice recognition accuracy, such as commands involving payment or security controls, can be disabled, while only basic and low-risk voice commands, such as playing music or checking the weather, can be allowed.

[0106] After identifying abnormal microphones in the microphone array, this application's solution does not solely rely on signal reconstruction to maintain system operation but also employs multiple countermeasures. First, by generating abnormality alerts, the system can promptly inform the user or relevant management modules of the microphone array's operational status, allowing the user to be aware of the current microphone array malfunction and take appropriate intervention measures, preventing unknowingly continued use of potentially defective equipment. Second, to ensure voice interaction quality as much as possible under abnormal conditions, this solution shortens the pickup distance of non-abnormal microphones, enabling the microphone array to focus more on capturing near-field voice signals and effectively suppressing far-field noise and reverberation interference. This allows for relatively clear near-field voice input even when part of the microphone array fails. Furthermore, based on the number of abnormal microphones, when the number reaches a certain threshold, the voice interaction device switches to near-field interaction mode. Finally, considering the potential negative impact of abnormal microphones on complex array noise reduction algorithms, this solution further switches to noise reduction operations on the audio sampling signals of individual non-abnormal microphones. This strategy avoids interference from erroneous information introduced by faulty microphones, ensuring that at least one microphone provides effectively noise-reduced speech signals, providing more reliable input for subsequent speech recognition and interaction. Furthermore, to prevent speech recognition errors caused by microphone array performance degradation, this solution also restricts the voice interaction device from performing preset target voice interaction operations. This restriction effectively reduces the risk of misrecognition and misoperation, protects user experience, and guides users to adopt more reliable interaction methods during microphone array malfunctions. These measures work together to form a comprehensive solution that effectively manages risk, optimizes performance, and enhances user experience when microphone array malfunctions.

[0107] The following is a concrete example. As a specific implementation, when the microphone array of a voice interaction device (such as a smart speaker) detects an anomaly in one of its microphones—for example, its signal power characteristics are significantly lower than other microphones—the system first generates an anomaly warning message. This message can be displayed by the smart speaker's LED indicator flashing red, accompanied by a voice announcement stating, "Microphone array detected an anomaly; some functions may be limited." Simultaneously, to address the performance degradation of the microphone array, the system automatically adjusts the pickup parameters of the non-abnormal microphones. For example, it might shorten the focusing range of the beamforming algorithm from 3 meters to 1.5 meters to ensure good sound pickup even when the user speaks close to the speaker. Furthermore, the system will stop using complex spatial noise reduction algorithms based on the entire array and instead select the non-abnormal microphone with the best signal quality in the array, applying a single-channel noise reduction algorithm based on a deep neural network to its audio sample signal to remove environmental noise. During this period, in order to avoid misoperations caused by a decrease in voice recognition accuracy, the system will restrict the smart speaker from performing certain operations that require high recognition accuracy, such as "opening the smart door lock" or "making online payments", but will still allow basic operations such as "playing music" or "checking the weather".

[0108] Through the above technical solutions, this application provides a more comprehensive and robust anomaly handling mechanism in the event of an abnormal microphone in the microphone array. Timely generation of anomaly alerts allows users to quickly understand the device status, preventing unknowingly continued use of potentially defective equipment, thereby improving user experience and device reliability. Shortening the pickup distance of non-abnormal microphones helps to effectively capture near-field speech even when some microphones fail, reducing far-field noise interference and ensuring the availability of core voice interaction functions. Switching to noise reduction for individual non-abnormal microphones avoids the negative impact of abnormal microphones on complex array noise reduction algorithms, ensuring that at least one microphone can provide high-quality voice input. Restricting the voice interaction device from performing preset target voice interaction operations effectively reduces the risk of misoperation due to decreased speech recognition accuracy, improving system security and stability. The combined application of these measures enables the voice interaction device to not only maintain basic functions through signal reconstruction when facing microphone anomalies, but also significantly improve system robustness, user experience, and security through proactive management and optimization.

[0109] See Figure 5 This application also provides a microphone anomaly handling device, which can implement the above-described microphone anomaly handling method. The device includes: The first module 501 is used to acquire audio sampling signals obtained from multiple microphones in the microphone array; The second module 502 is used to evaluate the target features of the audio sampled signal and obtain the feature evaluation results; the target features include the signal power features, signal energy features, signal-to-noise ratio features, cross-correlation features, coherence features and / or spectral features of the audio sampled signal; The third module 503 is used to identify abnormal microphones with signal anomalies based on the feature evaluation results; The fourth module 504 is used to reconstruct the audio signal at the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone and the location of the microphone, so as to obtain the audio reconstruction signal. The fifth module 505 is used to input the audio sampling signal and audio reconstruction signal of the non-abnormal microphone into the voice interaction device so that the voice interaction device can perform the corresponding voice interaction operation.

[0110] The specific implementation of this microphone anomaly handling device is basically the same as the specific embodiment of the microphone anomaly handling method described above, and will not be repeated here.

[0111] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0112] The following reference Figure 6 To describe an electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0113] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0114] The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the microphone abnormality handling method section of this specification according to various exemplary embodiments of this disclosure.

[0115] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0116] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0117] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0118] Electronic device 600 can also communicate with one or more external devices 600' (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0120] The microphone anomaly handling method, apparatus, device, and medium provided in this application, by evaluating the target features of the audio sampling signals of each microphone in a microphone array, can promptly and accurately identify abnormal microphones. Using the audio sampling signals of non-abnormal microphones and the position information of each microphone, the audio signal at the location of the abnormal microphone is reconstructed. Finally, the audio sampling signals of the non-abnormal microphones and the reconstructed audio signal are input into the voice interaction device, ensuring that the voice interaction device can continuously perform corresponding voice interaction operations. Therefore, by introducing feature evaluation and signal reconstruction mechanisms, the functional deficiencies of abnormal microphones are compensated for, enabling the voice interaction device to receive a complete and high-quality microphone array input, improving the robustness of the microphone array. Even if some microphones fail, the overall system can still maintain stable performance, avoiding damage to voice interaction functions due to microphone failure. It can adaptively handle microphone anomalies in the microphone array, improving the voice interaction experience.

[0121] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0122] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0124] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified to be uniquely different from one or more devices in this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0125] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A microphone anomaly handling method, characterized in that, include: Acquire audio sampling signals obtained from multiple microphones in a microphone array; The target features of the audio sampled signal are evaluated to obtain feature evaluation results; the target features include the signal power features, signal energy features, signal-to-noise ratio features, cross-correlation features, coherence features, and / or spectral features of the audio sampled signal; Based on the aforementioned feature evaluation results, an abnormal microphone with signal anomalies was identified; Based on the audio sampling signal of the non-abnormal microphone and the location of the microphone, the audio signal at the location of the abnormal microphone is reconstructed to obtain the audio reconstructed signal. The audio sampling signal from the non-abnormal microphone and the audio reconstruction signal are input into the voice interaction device so that the voice interaction device can perform corresponding voice interaction operations.

2. The microphone anomaly handling method according to claim 1, characterized in that, The evaluation of the target features of the audio sampled signal includes: Determine the feature deviation between the target feature and the reference feature of the audio sampling signal; the reference feature is the average value of the target features of other audio sampling signals or a preset target feature threshold. Determine whether the feature deviation value meets the preset feature deviation condition; If it does not meet the requirements, generate a feature evaluation result for the microphone anomaly; If the criteria are met, a feature evaluation result indicating that the microphone is not abnormal is generated.

3. The microphone anomaly handling method according to claim 1, characterized in that, The reconstruction of the audio signal at the location of the abnormal microphone, based on the audio sampling signal from the non-abnormal microphone and the microphone's location, includes: Based on the location of the non-abnormal microphones, reconstruct the effective geometric topology of the microphone array; Based on the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone, a signal reconstruction model corresponding to the location of the abnormal microphone is determined; the signal reconstruction model is used to characterize the mapping relationship between the audio sampling signal of the non-abnormal microphone and the expected audio signal at the location of the abnormal microphone. Based on the signal reconstruction model, the audio sampling signal of the non-abnormal microphone is processed to generate an audio reconstruction signal corresponding to the location of the abnormal microphone.

4. The microphone anomaly handling method according to claim 3, characterized in that, The signal reconstruction model is a model based on spatial sound field interpolation. The processing of the audio sampling signal from the non-abnormal microphone based on the signal reconstruction model includes: Based on the audio sampling signals of the non-abnormal microphones and the effective geometric topology, calculate the spatial correlation characteristics between the non-abnormal microphones; Based on the spatial correlation characteristics and the location of the abnormal microphone, the sound field characteristics at the location of the abnormal microphone are estimated by the signal reconstruction model. The audio reconstruction signal is generated based on the sound field characteristics at the location of the abnormal microphone.

5. The microphone anomaly handling method according to claim 3, characterized in that, The signal reconstruction model is a pre-trained neural network model. The processing of the audio sampling signal from the non-abnormal microphone based on the signal reconstruction model includes: The audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone are input into the signal reconstruction model; the signal reconstruction model is trained based on the audio sample signal, sample geometric topology, and sample abnormal location. Obtain the audio reconstruction signal output by the signal reconstruction model.

6. The microphone anomaly handling method according to claim 3, characterized in that, Before determining the signal reconstruction model corresponding to the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone, the effective geometric topology, and the location of the abnormal microphone, the method further includes: Based on the feature evaluation results, the device performance characteristics of the non-abnormal microphone are estimated; Based on the device performance characteristics of the non-abnormal microphone, signal compensation is performed on the audio sampling signal of the non-abnormal microphone to obtain the compensated audio sampling signal of the non-abnormal microphone.

7. The microphone anomaly handling method according to claim 1, characterized in that, After confirming the presence of the abnormal microphone, one or more of the following steps are also included: An abnormal prompt message is generated, the pickup distance of the non-abnormal microphone is shortened, and the voice interaction device is switched to near-field interaction mode according to the number of abnormal microphones. Switch to noise reduction operation on the audio sample signal of a single non-abnormal microphone; The voice interaction device is restricted from performing preset target voice interaction operations.

8. A microphone malfunction handling device, characterized in that, include: The first module is used to acquire audio sampling signals obtained from multiple microphones in the microphone array; The second module is used to evaluate the target features of the audio sampling signal and obtain the feature evaluation results; the target features include the signal power features, signal energy features, signal-to-noise ratio features, cross-correlation features, coherence features and / or spectral features of the audio sampling signal; The third module is used to identify abnormal microphones with signal anomalies based on the feature evaluation results. The fourth module is used to reconstruct the audio signal at the location of the abnormal microphone based on the audio sampling signal of the non-abnormal microphone and the location of the microphone, so as to obtain the audio reconstruction signal. The fifth module is used to input the audio sampling signal from the non-abnormal microphone and the audio reconstruction signal into the voice interaction device so that the voice interaction device can perform corresponding voice interaction operations.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the microphone abnormality handling method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the microphone anomaly handling method according to any one of claims 1 to 7.

Images