Intelligent camera audio parameter adaptive switching method and switching system

Abstract

The application discloses a kind of intelligent camera audio parameter adaptive switching method and system, including the following steps: obtaining the downlink audio stream received by intelligent camera, the downlink audio stream is decoded and frame processing, obtain multiple downlink audio frames, and allocate frame sequence number F_seq for each downlink audio frame, and based on uniform audio frame clock, the frame sequence number F_seq is associated with loudspeaker play reference frame and / or near-end microphone acquisition frame in corresponding time window;Calculate the decibel domain frame energy E_dB [n] of each downlink audio frame, and determine dynamic threshold T_h [k] according to downlink audio noise baseline N_F [k] and adaptive margin parameter M [k];According to the comparison result of the decibel domain frame energy E_dB [n] and the dynamic threshold T_h [k], combined with effective trigger count and suspend timer, the audio working state of the intelligent camera is judged.

Description

Technical Field

[0001] This invention relates to the field of smart camera technology, and in particular to a method and system for adaptive switching of audio parameters in a smart camera. Background Technology

[0002] Smart cameras typically feature remote voice intercom functionality, and their audio systems require audio parameter control in different operating states, including silent monitoring, one-way intercom, and two-way communication. In silent monitoring mode, the device usually needs high microphone sensitivity to capture distant or faint ambient sounds. In one-way intercom mode, the remote user sends downlink voice messages to the device via an app, and the device's speaker plays these messages. If the microphone maintains high gain, it can easily pick up the speaker's playback again, creating an echo. In two-way communication mode, both distant and local near-end voice messages coexist, and the coordination between microphone gain, acoustic echo cancellation parameters, and noise suppression parameters directly affects the two-way call quality.

[0003] In the prior art, Chinese patent application CN202010547493.6 and publication number CN111833900A disclose an audio gain control method, system, device, and storage medium. This method determines the current frame type by performing activity detection on each frame in the audio, and pushes the active frame into an array when the current frame is an active frame. Gain control is then performed based on the audio point amplitude and preset compressor parameters. While this solution involves audio frame detection and gain control, it primarily focuses on the automatic gain control of the audio signal itself and does not establish a three-state machine for silent monitoring, one-way intercom, and two-way communication modes for smart camera remote intercom scenarios. Furthermore, it does not disclose a technical means to synchronously correlate the current physical gain change with acoustic echo cancellation compensation parameters through frame sequence numbers when the audio codec's physical gain switches.

[0004] For example, Chinese patent application CN201710438854.1 and publication CN109036445A disclose a method for adjusting the audio source gain value of a microphone (MIC) sensor and an action camera. This method determines whether to reduce the current input audio source gain value of the MIC sensor to lower wind noise by considering the current acceleration and speed of the image acquisition mobile terminal, as well as the relationship between the orientation of the MIC sensor's inlet hole and the direction of movement. While this solution involves microphone gain adjustment in cameras or action cameras, its primary purpose is to adjust the microphone gain based on the motion state or wind noise risk, rather than performing a three-state mode discrimination based on downlink audio energy, local voice activity, and remote intercom status. Furthermore, this solution does not address the frame-level synchronization problem between the microphone's physical gain and the reference path of the acoustic echo cancellation module.

[0005] For example, Chinese patent application CN201911112801.6 and publication number CN111083297A discloses an echo cancellation method and electronic device. This method determines the presence of dual-talk by acquiring a first near-end signal, an estimated echo signal, and an error signal, and adjusts the working state of an adaptive filter based on the determination result to perform echo cancellation. While this solution improves echo cancellation processing in dual-talk scenarios, its focus is on adjusting the working state of the acoustic echo cancellation filter. It does not disclose the dynamic switching process of the PGA physical gain of the smart camera microphone, generating a gain synchronization message containing the current physical gain value, the corresponding audio frame sequence number, and a gain amplitude compensation factor. Nor does it disclose the technical means by which the acoustic echo cancellation module matches the frame sequence number and applies a gain amplitude compensation factor to the corresponding audio frame.

[0006] In addition, common audio parameter control methods in low-cost smart camera products include hardware voice activity detection solutions, third-party audio algorithm library solutions, fixed gain solutions, and simple threshold-triggered switching solutions. Among these, hardware voice activity detection solutions require the addition of dedicated detection chips or circuits, leading to increased hardware costs; third-party audio algorithm library solutions typically consume significant CPU and memory resources and may involve algorithm licensing costs; fixed gain or manual switching solutions struggle to adapt to real-time changes in call status; and simple threshold-triggered switching solutions are prone to false triggering or frequent switching due to transient noise, speech breaks, or increased ambient noise.

[0007] Meanwhile, in the existing remote intercom process of smart cameras, if the physical gain register of the audio codec is directly modified when the audio working state changes, it is easy to cause a sudden change in the audio waveform at non-zero crossing points, thus forming an audible pop. If the microphone physical gain has changed, but the acoustic echo cancellation module still estimates according to the reference path before the change, it will cause the echo path reference signal and the actual microphone pickup signal to mismatch in the energy domain, which will lead to an increase in echo residue, re-convergence of the acoustic echo cancellation module, and even affect the stability of two-way communication.

[0008] Therefore, existing smart camera audio parameter control solutions have at least the following shortcomings: First, some solutions rely on hardware voice activity detection modules or third-party audio algorithm libraries, which are costly and resource-intensive; second, some solutions do not combine the three working states of silent listening, one-way communication, and two-way talk in smart camera remote intercom for stable discrimination; third, fixed thresholds or simple energy triggering methods are difficult to adapt to different environmental noises and are prone to misjudgment due to transient noise or speech breaks; fourth, when the microphone physical gain changes, existing solutions fail to perform frame-level synchronization association of the current physical gain value, the corresponding audio frame sequence number, and the acoustic echo cancellation compensation parameters, resulting in inconsistency between the acoustic echo cancellation reference path and the actual signal picked up by the microphone in the energy domain during gain switching.

[0009] Therefore, there is an urgent need to provide an audio parameter adaptive switching method and system suitable for low-cost smart camera platforms, so as to achieve stable downlink audio state discrimination, smooth microphone gain switching, and frame-level synchronous compensation of acoustic echo cancellation parameters without adding hardware voice activity detection chips or relying on additional third-party voice activity detection modules, thereby improving the stability and audio quality during remote voice communication. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a method for adaptive switching of audio parameters in a smart camera.

[0011] To achieve the above objectives, the present invention provides a method for adaptive switching of audio parameters of a smart camera, comprising the following steps: Step S1: Obtain the downlink audio stream received by the smart camera, decode and frame the downlink audio stream to obtain multiple downlink audio frames, and assign a frame sequence number F_seq to each downlink audio frame. Based on a unified audio frame clock, associate the frame sequence number F_seq with the speaker playback reference frame and / or the near-end microphone acquisition frame in the corresponding time window. Step S2: Calculate the decibel-domain frame energy E_dB[n] of each downlink audio frame, and determine the dynamic threshold T_h[k] based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k]. Step S3: Based on the comparison result of the decibel domain frame energy E_dB[n] and the dynamic threshold T_h[k], combined with the effective trigger count and the suspension timer, determine the audio working state of the smart camera, and switch the audio working state between silent monitoring mode, one-way intercom mode and two-way call mode. Step S4: Determine the target gain value G_target based on the audio working state, and maintain the target gain value G_target and the current physical gain value G_current in the software layer; Step S5: When the current physical gain value G_current is inconsistent with the target gain value G_target, the current physical gain value G_current is updated step by step according to a preset step size, and at the time corresponding to the zero crossing point or near the zero crossing point, the updated current physical gain value G_current is written into the physical gain register of the audio codec. Step S6: After each step update of the current physical gain value G_current, a gain synchronization message is generated. The gain synchronization message includes the frame sequence number F_seq[k] corresponding to this step update, the current physical gain value G_current[k] after the step update, and the gain amplitude compensation factor α[k] calculated based on the current physical gain value G_current[k]. Step S7: Send the gain synchronization message to the acoustic echo cancellation module, so that when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], it applies the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to that audio frame, so that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain.

[0012] Preferably, in step S1, the downlink audio stream is decoded and framed, including: The downlink audio stream is decoded to obtain a downlink PCM audio stream; The downlink PCM audio stream is divided into frames with a frame length of 10ms to 20ms to obtain multiple downlink audio frames; Each downlink audio frame is assigned a frame sequence number F_seq using an incremental counting method.

[0013] Preferably, in step S2, the decibel-domain frame energy E_dB[n] of each downlink audio frame is calculated, and the dynamic threshold T_h[k] is determined based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k], including: The RMS energy E_RMS[n] of the nth downlink audio frame is calculated according to the formula E_RMS[n]=sqrt((1 / L)×Σx_i²), where L is the number of sampling points in the downlink audio frame and x_i is the amplitude of the i-th sampling point in the downlink audio frame; the RMS energy E_RMS[n] is then converted into decibel-domain frame energy E_dB[n]. The downlink audio noise baseline N_F[k] is updated based on the low-energy frames in the decibel domain frame energy E_dB[n] of multiple historical downlink audio frames; The dynamic threshold T_h[k] is determined according to the formula T_h[k]=N_F[k]+M[k]; wherein, N_F[k], M[k] and T_h[k] are all decibel-domain parameters; When the downlink audio noise baseline N_F[k] is higher than the high noise environment judgment threshold T_noisy, the adaptive margin parameter M[k] is adjusted towards the upper limit M_max. When the downlink audio noise baseline N_F[k] is lower than the quiet environment judgment threshold T_quiet, the adaptive margin parameter M[k] is adjusted in the direction of the lower limit M_min; The downlink audio noise baseline N_F[k] is taken as the moving average of low-energy frames in the most recent 80 to 120 frames; the upper limit M_max of the adaptive margin parameter M[k] is 10dB, the lower limit M_min is 6dB, and the change of the adaptive margin parameter M[k] every 100ms does not exceed 1dB.

[0014] Preferably, in step S3, determining the audio operating status of the smart camera includes: When the decibel frame energy E_dB[n] of N consecutive downlink audio frames is greater than the dynamic threshold T_h[k], a valid downlink voice signal is determined to exist; when the decibel frame energy E_dB[n] of the downlink audio frame is lower than the dynamic threshold T_h[k], the suspension timer is started; when the suspension timer reaches the preset suspension time T_hangover, the valid downlink voice signal is determined to have disappeared; when the decibel frame energy E_dB[n] of the downlink audio frame is greater than the dynamic threshold T_h[k] again, the suspension timer is reset; where N is 3 to 5, and T_hangover is 5. 00ms~1s; Based on the decibel-domain frame energy E_dB[n] and the downlink audio noise baseline N_F[k], the signal-to-noise ratio SNR[k] is calculated according to SNR[k]=E_dB[n]-N_F[k]; When the decibel-domain frame energy E_dB[n] is greater than the dynamic threshold T_h[k] and the signal-to-noise ratio SNR[k] is lower than the preset signal-to-noise ratio threshold SNR_th, the effective trigger count is not accumulated, and the downlink audio frame is not used to update the downlink audio noise baseline N_F[k]; The downlink audio noise baseline N_F[k] is updated according to the low-energy frame.

[0015] Preferably, in step S3, switching the audio operating state between silent monitoring mode, one-way intercom mode, and two-way communication mode includes: In silent monitoring mode, when a valid downlink voice signal is detected, switch to one-way intercom mode; In one-way intercom mode, when continuous local voice activity is detected, switch to two-way call mode; In two-way call mode, when it is determined that the local continuous voice activity has disappeared and the downlink valid voice signal still exists, switch to one-way intercom mode. In one-way intercom mode or two-way call mode, when it is determined that the downlink valid voice signal has disappeared, switch to silent monitoring mode; The determination of the existence of local continuous speech activity includes: calculating the local RMS energy E_local[n] of the near-end audio frames captured by the local microphone; comparing the local RMS energy E_local[n] with the local speech detection threshold T_local; determining the existence of local continuous speech activity when the local RMS energy E_local[n] of N consecutive near-end audio frames is greater than the local speech detection threshold T_local; wherein, N_local is 2 to 4.

[0016] Preferably, in step S4, determining the target gain value G_target based on the audio operating state includes: In silent monitoring mode, the target gain value G_target is determined as the first gain value; In one-way intercom mode, the target gain value G_target is determined as the second gain value; In two-way communication mode, the target gain value G_target is determined as the third gain value; Wherein, the first gain value is greater than the second gain value, and the second gain value is greater than the third gain value.

[0017] Preferably, in step S5, the current physical gain value G_current is updated in steps according to a preset step size, and at the time corresponding to the zero-crossing point or near-zero-crossing point, the updated current physical gain value G_current is written into the physical gain register of the audio codec, specifically including: The current physical gain value G_current is updated by incrementing or decrementing in steps no greater than 1.5dB. A step update is performed every 10ms to 20ms until the current physical gain value G_current reaches the target gain value G_target. When the audio codec supports hardware zero-crossing detection, the hardware zero-crossing detection function is enabled, and when the zero-crossing point of the audio signal is detected, the current physical gain value G_current after step update is written to the physical gain register. When the audio codec does not support hardware zero-crossing detection, software zero-crossing detection is performed on the PCM audio data, and when the software detects a zero-crossing point or near-zero-crossing point, the updated physical gain value G_current is written to the physical gain register. The software zero-crossing detection of PCM audio data includes: calculating the sampling mean μ of the current audio frame; subtracting the sampling mean μ from the amplitude of each sampling point in the current audio frame to obtain the sampling sequence x'[i] after DC bias removal; determining that there is a zero-crossing point between x'[i] and x'[i+1] when x'[i]×x'[i+1]≤0; and determining that there is an approximate zero-crossing point when no zero-crossing point is detected, and the sampling time that satisfies |x'[i]|<ε; where ε is 0.5% to 1% of the full-scale amplitude of the analog-to-digital converter. In step S5, clipping status is also detected for the local microphone signal. The detection of clipping status includes: determining whether the number of sampling points that reach or approach the full-scale amplitude of the analog-to-digital converter in a consecutive preset number of audio frames exceeds a preset number threshold; when the preset number threshold is exceeded, clipping is determined to have occurred, the step update is skipped, and the current physical gain value G_current is reduced by a preset attenuation amount in one go.

[0018] Preferably, in step S6, the gain amplitude compensation factor α[k] is calculated according to the following formula: α[k]=10^(G_current[k] / 20)×α_base; Where G_current[k] is the current physical gain value after the k-th step update, and α_base is the base compensation coefficient; The gain synchronization message also includes a step direction identifier and a timestamp; the step direction identifier is used to indicate whether the current physical gain value G_current is in a gain reduction phase, a gain increase phase, or a stable phase; the timestamp is used to reset or rebuild the gain synchronization message queue when the frame sequence number changes. When the current physical gain value G_current is reduced by a preset attenuation amount in one go, in step S6, a corresponding gain synchronization message is generated for the current physical gain value G_current after the one-time reduction.

[0019] Preferably, in step S7, the gain synchronization message is sent to the acoustic echo cancellation module, so that when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], it applies the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to that audio frame, specifically including: S71. Write the gain synchronization message into the gain synchronization message queue; S72. The acoustic echo cancellation module queries the gain synchronization message queue according to the frame sequence number of the audio frame to be processed; S73. When the frame sequence number of the audio frame to be processed is consistent with F_seq[k] in the gain synchronization message, read the gain amplitude compensation factor α[k] corresponding to F_seq[k]. S74. Before processing the audio frame to be processed, read and latch the gain amplitude compensation factor α[k] corresponding to the audio frame to be processed, keep the gain amplitude compensation factor α[k] unchanged before the processing of the audio frame to be processed ends, and apply it to the echo path reference signal corresponding to the audio frame to be processed to adjust the amplitude of the echo path reference signal.

[0020] When the current physical gain value G_current is reduced by a preset attenuation amount at once, in step S7, the gain synchronization message corresponding to the reduced current physical gain value G_current is sent to the acoustic echo cancellation module. During the step update of the current physical gain value G_current towards a lower gain direction, the nonlinear processing level in the acoustic echo cancellation module is increased; when the current physical gain value G_current reaches the target gain value G_target, the nonlinear processing level is restored.

[0021] This invention provides an intelligent camera audio parameter adaptive switching system, comprising: The audio framing module is used to acquire the downlink audio stream received by the smart camera, decode and framing the downlink audio stream to obtain multiple downlink audio frames, and assign a frame sequence number F_seq to each downlink audio frame. Based on a unified audio frame clock, the frame sequence number F_seq is associated with the speaker playback reference frame and / or the near-end microphone acquisition frame within the corresponding time window. The energy threshold module is used to calculate the decibel-domain frame energy E_dB[n] of each downlink audio frame and determine the dynamic threshold T_h[k] based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k]. The state discrimination module is used to determine the audio working state of the smart camera based on the comparison result of the decibel domain frame energy E_dB[n] and the dynamic threshold T_h[k], combined with the effective trigger count and the suspension timer, so that the audio working state can be switched between silent monitoring mode, one-way intercom mode and two-way call mode. The gain mapping module is used to determine the target gain value G_target according to the audio working state; the shadow gain control module is used to maintain the target gain value G_target and the current physical gain value G_current in the software layer, and update the current physical gain value G_current in steps according to a preset step size when the current physical gain value G_current is inconsistent with the target gain value G_target. The register writing module is used to write the updated physical gain value G_current to the physical gain register of the audio codec at the moment corresponding to the zero crossing or near zero crossing. The gain synchronization module is used to generate a gain synchronization message after each step update of the current physical gain value G_current. The gain synchronization message includes the frame sequence number F_seq[k] corresponding to this step update, the current physical gain value G_current[k] after the step update, and the gain amplitude compensation factor α[k] calculated based on the current physical gain value G_current[k]. The acoustic echo cancellation linkage module is used to receive the gain synchronization message and, when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], apply the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to the audio frame so that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain. The register writing module includes a hardware zero-crossing writing unit and a software zero-crossing writing unit. The hardware zero-crossing writing unit is used to enable the hardware zero-crossing detection function when the audio codec supports it, and write to the physical gain register when a zero-crossing point of the audio signal is detected. The software zero-crossing writing unit is used to perform mean elimination processing on the PCM audio data to obtain a sampled sequence after DC bias removal when the audio codec does not support the hardware zero-crossing detection function, and find the zero-crossing point or near-zero-crossing point according to the sampled sequence, and write to the physical gain register at the time corresponding to the zero-crossing point or the near-zero-crossing point. The acoustic echo cancellation linkage module includes a message queue unit and a frame matching unit. The message queue unit is used to cache the gain synchronization message. The frame matching unit is used to query the message queue unit according to the frame sequence number of the audio frame to be processed, and when the frame sequence number of the audio frame to be processed is consistent with F_seq[k] in the gain synchronization message, it reads the gain amplitude compensation factor α[k] corresponding to F_seq[k], and applies the gain amplitude compensation factor α[k] during the amplitude update process of the echo path reference signal of the audio frame to be processed.

[0022] The technical solution of this invention has the following beneficial effects: This invention performs frame-by-frame processing on the downlink audio stream using software, calculates the RMS energy frame by frame, and generates a dynamic threshold T_h[k] by combining the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k]. This enables downlink audio energy detection and audio working state discrimination without adding a hardware voice activity detection chip or relying on additional third-party voice activity detection modules, thereby reducing hardware BOM costs, algorithm licensing costs, and system resource consumption. At the same time, by dynamically adjusting the adaptive margin parameter M[k] according to the noise level in the downlink audio stream, the device can increase the trigger threshold to reduce false triggers when the downlink audio stream noise level is high, and decrease the trigger threshold to improve voice detection sensitivity when the downlink audio stream noise level is low, thus enhancing the adaptability under different downlink audio noise conditions.

[0023] This invention employs a dual anti-jitter mechanism consisting of effective trigger counting and a suspension timer. Only when multiple consecutive frames meet the dynamic threshold condition is downlink valid speech determined to exist. During brief pauses in speech, the current state is maintained through a suspension time, thereby reducing frequent mode jumps caused by transient noise, speech fragmentation, or network audio jitter. Simultaneously, when the decibel-domain frame energy E_dB[n] of the downlink audio frame exceeds the dynamic threshold but the signal-to-noise ratio is low, the effective trigger count is not accumulated, and the corresponding audio frame is not used to update the downlink audio noise baseline. This suppresses false triggers caused by low signal-to-noise ratio background noise in the downlink audio stream, improving the accuracy and stability of audio state switching.

[0024] This invention uses a three-state machine to divide the audio working state of a smart camera into silent monitoring mode, one-way intercom mode, and two-way call mode, and maps different target gain values ​​according to different audio working states. This enables the device to automatically switch audio parameters between silent sound pickup, remote intercom, and two-way call, taking into account the sound pickup sensitivity in silent mode, the echo suppression capability in one-way intercom mode, and the call stability in two-way call mode.

[0025] This invention maintains a target gain value G_target and a current physical gain value G_current at the software layer. When the target gain value changes, it does not directly jump to write the physical gain register of the audio codec. Instead, it updates the current physical gain value step by step according to a preset step size, so that the physical gain gradually approaches the target gain value. At the same time, when writing to the physical gain register, it combines hardware zero-crossing detection or software zero-crossing detection to ensure that the current physical gain value is written to the physical gain register of the audio codec as close to the zero-crossing point of the audio signal as possible. This reduces the volume break, audible popping caused by sudden gain changes, and the impact of register writing on the continuity of the audio waveform, thereby improving the listening smoothness during gain switching.

[0026] After each step update of the current physical gain value G_current, this invention generates a gain synchronization message containing the frame sequence number F_seq[k], the current physical gain value G_current[k], and the gain amplitude compensation factor α[k], and sends it to the acoustic echo cancellation module. The acoustic echo cancellation module matches the corresponding gain amplitude compensation factor according to the frame sequence number of the audio frame to be processed, and applies the gain amplitude compensation factor to the amplitude of the echo path reference signal of the frame in a single, uninterrupted operation. This ensures that the echo path reference signal and the actual microphone pickup signal are linearly aligned in the amplitude domain, reducing the impact of operating system scheduling delay, audio thread delay, or message queue delay on the accuracy of AEC synchronization compensation, and reducing echo remnants during gain switching and the probability of the acoustic echo cancellation module reconverging.

[0027] This invention achieves frame-level synchronization between shadow register step control and acoustic echo cancellation gain amplitude compensation factor, enabling the acoustic echo cancellation module to obtain gain compensation information matching the current processing frame during dynamic adjustment of the microphone physical gain. This alleviates the problem of mismatch between AEC reference signal and actual pickup signal caused by dynamic gain adjustment, and achieves a balance between dynamic gain adjustment and echo cancellation stability.

[0028] When clipping is detected in a preset number of consecutive audio frames, this invention can skip the conventional stepping process and reduce the current physical gain value by a preset attenuation amount in one go. At the same time, it generates a corresponding gain synchronization message and sends it to the acoustic echo cancellation module. This allows for rapid reduction of analog gain when the microphone input is overloaded, and enables AEC compensation to follow the gain change in a timely manner, thereby reducing the impact of clipping distortion on the echo cancellation effect.

[0029] In two-way communication mode, this invention can reduce the automatic noise suppression level and temporarily increase the nonlinear processing level while the current physical gain value is adjusted towards a lower gain direction. This reduces the probability of near-end speech being over-suppressed and enhances the ability to suppress short-term residual echo, thereby improving the naturalness of speech and call stability during remote intercom between smart cameras. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the overall process of the intelligent camera audio parameter adaptive switching method of the present invention. Figure 2 This is a schematic diagram of the state transition of the three-state machine of the present invention; Figure 3 This is a timing diagram illustrating the convergence state of the step-by-step gain smoothing switching and acoustic echo cancellation based on the shadow register of the present invention. Figure 4This is a timing diagram illustrating the frame-level latch synchronization of the gain amplitude compensation factor in the acoustic echo cancellation linkage protection mechanism of the present invention. Figure 5 This is a flowchart of the clipping detection and emergency gain reduction process of the present invention; Figure 6 This is a structural block diagram of the intelligent camera audio parameter adaptive switching system of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Figures 1 to 6 This embodiment provides a method for adaptive switching of audio parameters in a smart camera, applicable to embedded audio-visual devices with remote intercom functionality, such as smart cameras, network cameras, doorbell cameras, and surveillance cameras. The smart camera includes a processor, an audio codec, a microphone, a speaker, a memory, and an acoustic echo cancellation module running within the processor. The processor receives downlink audio streams from remote devices and adaptively switches microphone gain, acoustic echo cancellation parameters, and related audio processing parameters based on changes in the energy of the downlink audio stream and local voice activity.

[0033] This embodiment is particularly suitable for low-cost SoC platforms that do not have a hardware voice activity detection module and do not rely on additional third-party voice activity detection modules. By performing energy detection, dynamic threshold judgment, state switching, gain smoothing control, and AEC synchronization compensation on downlink audio frames in software, adaptive switching of audio parameters can be achieved without increasing additional hardware costs.

[0034] In this embodiment, the smart camera can operate in silent monitoring mode, one-way intercom mode, and two-way communication mode. Silent monitoring mode refers to the smart camera maintaining high microphone sensitivity to capture ambient sound when there is no effective voice input from the remote end or the downlink audio does not meet the conditions for effective voice input. One-way intercom mode refers to the smart camera reducing microphone gain after detecting effective downlink voice from the remote end to reduce echoes from the speaker. Two-way communication mode refers to the smart camera further reducing microphone gain and working with the acoustic echo cancellation module for synchronous compensation when both remote downlink voice and local near-end voice are present, ensuring echo suppression and voice clarity during two-way communication.

[0035] The method in this embodiment includes the following steps.

[0036] Step S1: The smart camera acquires the received downlink audio stream, decodes and segments the downlink audio stream to obtain multiple downlink audio frames, and assigns a frame sequence number F_seq to each downlink audio frame. Based on a unified audio frame clock, a frame sequence number association is established between the downlink audio frames and the speaker playback reference frame / near-end acquisition frame within the corresponding time window.

[0037] Specifically, the downlink audio stream sent by the remote terminal can be a compressed and encoded audio stream. After receiving the downlink audio stream, the smart camera first decodes it into a downlink PCM audio stream through the audio decoding module. The downlink PCM audio stream can be a mono or stereo audio stream, with a sampling rate of 8kHz, 16kHz, 32kHz, or 48kHz, and a sampling bit width of 16bit or 24bit.

[0038] The processor divides the downlink PCM audio stream into frames according to a preset frame length, preferably 10ms to 20ms. For example, with a sampling rate of 16kHz and a frame length of 20ms, each downlink audio frame includes 320 sampling points. The processor assigns a unique, incrementing frame sequence number F_seq to each downlink audio frame. The frame sequence number F_seq can be implemented using an unsigned integer counting method, such as a uint32 type, and resets to zero when the count overflows. By assigning a frame sequence number F_seq to each downlink audio frame, a correspondence can be established between subsequent gain change information and specific audio frames, thus facilitating synchronous compensation by the acoustic echo cancellation module according to the frame sequence number.

[0039] In one specific embodiment, the frame length is 10ms, the effective trigger count threshold N is 4 frames, the hangover time T_hangover is 720ms, the initial value of the adaptive margin parameter M[0] is 8dB, the lower limit of the adaptive margin parameter M_min is 6dB, the upper limit of the adaptive margin parameter M_max is 10dB, the step size ΔG is 1.5dB, and the step interval Δt is 20ms; the silent monitoring mode corresponds to a PGA gain of 30dB, the one-way intercom mode corresponds to a PGA gain of 18dB, and the two-way call mode corresponds to a PGA gain of 6dB. The above parameters are only one specific embodiment and do not constitute a limitation on the scope of protection of this invention.

[0040] Step S2: Calculate the decibel-domain frame energy E_dB[n] of each downlink audio frame, and determine the dynamic threshold T_h[k] based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k].

[0041] Specifically, for the nth downlink audio frame, the processor calculates the RMS energy of that downlink audio frame using the following formula: E_RMS[n]=sqrt((1 / L)×Σx_i²) Where E_RMS[n] represents the RMS energy of the nth downlink audio frame, L represents the number of sampling points in the downlink audio frame, and x_i represents the amplitude of the i-th sampling point in the downlink audio frame. After obtaining the RMS energy E_RMS[n], the processor converts the RMS energy E_RMS[n] into decibel-domain frame energy E_dB[n]. For example, the decibel-domain frame energy E_dB[n] can be calculated according to the preset reference amplitude E_ref, using E_dB[n] = 20log10(E_RMS[n] / E_ref). The subsequent downlink audio noise baseline N_F[k], adaptive margin parameter M[k], and dynamic threshold T_h[k] are all represented in the decibel domain and compared based on the decibel-domain frame energy E_dB[n].

[0042] The processor further updates the downlink audio noise baseline N_F[k] based on low-energy frames in the decibel domain frame energy E_dB[n] of multiple historical downlink audio frames. For example, a moving average can be performed on low-energy frames from the most recent 80 to 120 frames to obtain the current downlink audio noise baseline N_F[k]. Low-energy frames can be understood as audio frames whose energy is in the lower range of the historical energy distribution, used to characterize the background noise level in the current downlink audio stream. In a specific embodiment, the moving average of low-energy frames from the most recent 100 frames can be taken as the downlink audio noise baseline N_F[k].

[0043] The dynamic threshold T_h[k] is determined according to the following formula: T_h[k]=N_F[k]+M[k], where N_F[k] is the downlink audio noise baseline obtained in the kth update, and M[k] is the adaptive margin parameter.

[0044] In one embodiment, when the downlink audio noise baseline N_F[k] is higher than the high-noise environment judgment threshold T_noisy, it indicates that the background noise level in the current downlink audio stream is high. The processor adjusts the adaptive margin parameter M[k] towards the upper limit M_max to avoid the background noise in the downlink audio stream being misjudged as valid far-end speech. When the downlink audio noise baseline N_F[k] is lower than the quiet environment judgment threshold T_quiet, it indicates that the background noise level in the current downlink audio stream is low. The processor adjusts the adaptive margin parameter M[k] towards the lower limit M_min to improve the sensitivity of far-end speech detection. In a specific embodiment, the initial value of M[k] is 8dB, M_max is 10dB, M_min is 6dB, and the change in the adaptive margin parameter M[k] every 100ms does not exceed 1dB, thereby avoiding frequent state jitter caused by dynamic threshold abrupt changes.

[0045] Step S3: Based on the comparison result of the decibel domain frame energy E_dB[n] and the dynamic threshold T_h[k], combined with the effective trigger count and the suspend timer, determine the audio working state of the smart camera, and switch the audio working state between silent monitoring mode, one-way intercom mode and two-way call mode.

[0046] Specifically, when the decibel-domain frame energy E_dB[n] of N consecutive downlink audio frames is greater than the dynamic threshold T_h[k], the processor determines that a valid downlink voice signal exists, where N is preferably 3 to 5. In a specific embodiment, N is 4, meaning that a valid downlink voice signal is determined to exist only when four consecutive frames satisfy the condition that the decibel-domain frame energy E_dB[n] is greater than the dynamic threshold T_h[k]. By judging through multiple consecutive frames, false triggering caused by instantaneous noise, impulse sounds, or network audio jitter in a single frame can be avoided.

[0047] When the decibel frame energy E_dB[n] of a certain downlink audio frame is lower than the dynamic threshold T_h[k], the processor does not immediately determine that the valid downlink voice signal has disappeared, but instead starts a suspension timer. The processor determines that the valid downlink voice signal has disappeared only when the suspension timer reaches a preset suspension time T_hangover. The preset suspension time T_hangover is preferably 500ms to 1s. In a specific embodiment, T_hangover is set to 720ms. When the decibel frame energy E_dB[n] of the downlink audio frame is greater than the dynamic threshold T_h[k] again during the suspension timer's operation, the processor resets the suspension timer. This suspension mechanism avoids frequent switching of audio operating states caused by short pauses in speech.

[0048] In this embodiment, the switching logic of the audio working state is as follows: In silent monitoring mode, when the processor determines that there is a valid downlink voice signal, it switches to one-way intercom mode; in one-way intercom mode, when the processor determines that there is a continuous local voice activity, it switches to two-way communication mode; in two-way communication mode, when the processor determines that the continuous local voice activity has disappeared and the valid downlink voice signal still exists, it switches to one-way intercom mode; in one-way intercom mode or two-way communication mode, when the processor determines that the valid downlink voice signal has disappeared, it switches to silent monitoring mode.

[0049] The method for determining local continuous voice activity can be as follows: the processor calculates the local RMS energy E_local[n] for the near-end audio frames captured by the local microphone, and compares the local RMS energy E_local[n] with the local voice detection threshold T_local; when the local RMS energy E_local[n] of N consecutive near-end audio frames is greater than the local voice detection threshold T_local, local continuous voice activity is determined to exist. N_local is preferably 2 to 4. When the local RMS energy is lower than T_local, the local voice detection counter is cleared to zero; when exiting the two-way call mode, the local voice detection counter can be reset and re-enter the suspension timing judgment process.

[0050] The state transition conditions for a three-state machine can be shown in Table 1.

[0051] Table 1

[0052] Among them, the effective trigger counter C is cleared every time E_dB[n] is lower than T_h[k]; the T_hangover timer is reset every time E_dB[n] exceeds T_h[k] again; and the local voice counter N_local is cleared when the local RMS energy is lower than T_local.

[0053] In a further embodiment, to avoid false triggering caused by low signal-to-noise ratio (SNR) noise in the downlink audio stream, the processor calculates the SNR[k] based on the decibel-domain frame energy E_dB[n] and the downlink audio noise baseline N_F[k], according to SNR[k] = E_dB[n] - N_F[k]. When the decibel-domain frame energy E_dB[n] is greater than the dynamic threshold T_h[k] but the SNR[k] is lower than the preset SNR threshold SNR_th, the processor does not accumulate the valid trigger count and does not use the downlink audio frame to update the downlink audio noise baseline N_F[k]; the downlink audio noise baseline N_F[k] is still updated based on historical low-energy frames. The preset SNR threshold SNR_th can be set according to the device application scenario. This reduces false triggering caused by increased background noise in the downlink audio stream.

[0054] Step S4: Determine the target gain value G_target based on the audio working status, and maintain the target gain value G_target and the current physical gain value G_current in the software layer.

[0055] Specifically, the processor pre-sets target gain values ​​corresponding to different audio operating states. Silent monitoring mode corresponds to the first gain value, one-way intercom mode corresponds to the second gain value, and two-way communication mode corresponds to the third gain value. The first gain value is greater than the second gain value, and the second gain value is greater than the third gain value.

[0056] For example, in one specific embodiment, the first gain value corresponding to the silent monitoring mode can be 30dB, the second gain value corresponding to the one-way intercom mode can be 18dB, and the third gain value corresponding to the two-way communication mode can be 6dB. The above values ​​are merely examples and do not constitute a limitation of the present invention. In practical applications, adjustments can be made based on microphone sensitivity, speaker sound pressure level, device structure cavity, and target pickup distance.

[0057] The processor maintains the target gain value G_target and the current physical gain value G_current at the software layer. The target gain value G_target is the desired gain value determined based on the current audio operating state; the current physical gain value G_current is the actual gain value that has been written to the audio codec's physical gain register. By distinguishing between the target gain value and the current physical gain value, a smooth transition in gain can be achieved, rather than abruptly changing during state transitions.

[0058] In some embodiments, the target gain value may also be referred to as the shadow gain value or the software-layer target gain register value G_shadow; G_shadow is used to quickly respond to audio operating state switching, and the current physical gain value G_current is used to characterize the gain value actually written to the audio codec's physical gain register. G_shadow can jump during state switching, while G_current gradually approaches G_shadow in preset steps.

[0059] In some embodiments, when the speaker volume is at a preset high volume level, the processor also locks the microphone PGA gain limit to a preset safety threshold. When the target gain value G_target, determined based on the audio operating state, is greater than this preset safety threshold, the processor uses the preset safety threshold as the actual target gain value. This method can reduce the risk of feedback and echo saturation when the speaker is playing at high volume.

[0060] Step S5: When the current physical gain value G_current is inconsistent with the target gain value G_target, the processor updates the current physical gain value G_current in steps according to a preset step size, and writes the updated current physical gain value G_current into the physical gain register of the audio codec at the zero-crossing point or near-zero-crossing point.

[0061] Specifically, when G_current is greater than G_target, the processor gradually decreases G_current by a preset step size; when G_current is less than G_target, the processor gradually increases G_current by a preset step size. The preset step size is preferably no greater than 1.5dB, and an update is performed every 10ms to 20ms until G_current reaches G_target. In one specific embodiment, the preset step size ΔG is 1.5dB, and the step interval Δt is 20ms. By using small steps and multiple updates, popping sounds, breaks, or noticeable volume jumps caused by sudden gain changes can be avoided.

[0062] In one embodiment, when the audio codec supports hardware Zero-Crossing, the processor enables the hardware zero-crossing detection function of the audio codec, and writes the updated G_current to the physical gain register when the hardware detects a zero-crossing of the audio signal. Since the amplitude of the audio signal is small near the zero-crossing point, writing to the physical gain register at this moment can reduce the impact of register switching on the continuity of the audio waveform.

[0063] In another embodiment, when the audio codec does not support hardware zero-crossing, the processor performs software zero-crossing detection on the PCM audio data. Specifically, the processor calculates the sampling mean μ of the current audio frame, subtracts the sampling mean μ from the amplitude of each sampling point in the current audio frame, and obtains the DC-bias-free sampling sequence x'[i], i.e., x'[i] = x[i] - μ. When x'[i] × x'[i+1] ≤ 0, it is determined that there is a zero-crossing between x'[i] and x'[i+1]. In some embodiments, the sampling time of the smaller of |x'[i]| and |x'[i+1]| can be selected as the approximate zero-crossing point, and the physical gain register is written. When no zero-crossing point is detected in the current audio frame, the sampling time that satisfies |x'[i]| < ε is determined as the approximate zero-crossing point, where ε can be 0.5% to 1% of the full-scale amplitude of the analog-to-digital converter. When the processor detects a zero-crossing point or an approximate zero-crossing point, it writes the updated G_current to the physical gain register. By using software zero-crossing detection, low-cost codec or SoC platforms that do not have hardware zero-crossing capabilities can still achieve gain switching with low popping sounds.

[0064] In a further embodiment, the processor also detects clipping status for the local microphone signal. Specifically, the processor determines whether the number of sampling points reaching or approaching the full-scale amplitude of the analog-to-digital converter in a consecutive preset number of audio frames exceeds a preset threshold; when the preset threshold is exceeded, clipping is determined to have occurred in the local microphone signal. At this time, the processor skips the above-mentioned step update process and reduces the current physical gain value G_current by a preset attenuation amount in one go. Preferably, the consecutive preset number can be 3 frames, and the preset attenuation amount can be 6dB. Through this clipping emergency processing, the microphone analog gain can be quickly reduced when the input signal is overloaded, preventing the clipping from continuing to expand.

[0065] like Figure 3 As shown, Figure 3 This is a timing diagram illustrating the convergence state of the step-by-step gain smoothing and acoustic echo cancellation based on the shadow register. Figure 3 This includes gain variation timing, echo cancellation enhancement variation timing, and zero-crossing detection trigger timing arranged along the same time axis.

[0066] In the gain change timing, the software-layer target gain value G_target or shadow gain value G_shadow can change abruptly at state transition times. For example, at time t0, the audio operating state switches from silent monitoring mode to one-way intercom mode, and G_shadow changes from 30dB to 18dB; at time t1, the audio operating state switches from one-way intercom mode to two-way communication mode, and G_shadow changes from 18dB to 6dB. Unlike G_shadow, the current physical gain value G_current does not jump directly to the target value, but rather approaches the target gain value step by step according to a preset step size. For example, G_current can gradually transition from 30dB to 18dB by decreasing by 1.5dB every 20ms, and then gradually transition from 18dB to 6dB during subsequent state transitions.

[0067] Figure 3 The document also illustrates a one-time gain transition method for comparison. In this method, the physical gain value jumps directly from the original gain value to the target gain value at the moment of state switching, which can easily cause abrupt changes in the audio waveform and mismatch of the echo cancellation reference signal. In contrast, this embodiment separates G_shadow and G_current, enabling the software layer to respond quickly to state switching, while allowing the physical gain register to change smoothly in steps, thereby reducing audible pops and echo remnants during gain switching.

[0068] In the timing of echo cancellation enhancement changes, this embodiment generates a gain synchronization message containing the frame sequence number F_seq, the current physical gain value G_current, and the gain amplitude compensation factor α[k] after each G_current step update, and sends this gain synchronization message to the acoustic echo cancellation module. When processing audio frames matching F_seq, the acoustic echo cancellation module applies the corresponding α[k], ensuring that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain. Therefore, during gain step changes, the acoustic echo cancellation module can maintain a relatively stable convergence state.

[0069] In the zero-crossing detection timing, each write to the physical gain register is performed as close as possible to the zero-crossing point or near the zero-crossing point of the audio signal. Specifically, when the audio codec supports hardware zero-crossing detection, the processor waits for hardware zero-crossing detection to trigger before writing to the physical gain register; when the audio codec does not support hardware zero-crossing detection, the processor determines the zero-crossing point or near the zero-crossing point through software zero-crossing detection and writes to the physical gain register at the corresponding time. This approach reduces the impact of gain writing on the continuity of the audio waveform.

[0070] In one test case, compared to a one-time gain jump method, this embodiment can reduce the drop in echo cancellation performance during gain switching and shorten the reconvergence process of the acoustic echo cancellation module.

[0071] Step S6: After each step update of the current physical gain value G_current, the processor generates a gain synchronization message.

[0072] The gain synchronization message includes the frame sequence number F_seq[k] corresponding to this step update, the current physical gain value G_current[k] after the step update, and the gain amplitude compensation factor α[k] calculated based on the current physical gain value G_current[k].

[0073] The gain magnitude compensation factor α[k] is calculated according to the following formula: α[k]=10^(G_current[k] / 20)×α_base Where G_current[k] is the current physical gain value after the k-th step update, and α_base is the base compensation coefficient.

[0074] In this embodiment, the gain synchronization message may further include a step direction identifier and a timestamp. The step direction identifier is used to indicate whether the current physical gain value G_current is in a gain reduction phase, a gain increase phase, or a stable phase; the timestamp is used to reset or rebuild the gain synchronization message queue when the frame sequence number changes or the message queue is abnormal.

[0075] In one specific embodiment, the processor maintains a gain information structure of the following form: struct gain_info{ uint32 F_seq; / / The audio frame sequence number corresponding to the current gain change float G_current; / / Current PGA gain value float alpha; / / Gain magnitude compensation factor α[k] enum ramp_state; / / Current ramp direction, including RAMP_DOWN, RAMP_UP, and STABLE uint64 timestamp; / / Frame clock timestamp } The message is sent via a callback function within the same CPU execution cycle after the gain step is written to the physical register. The acoustic echo cancellation module uses a message queue to cache the most recent N_queue gain synchronization messages. The value of N_queue can be 8 to 16, and it is matched using the frame sequence number F_seq as the key. When processing the corresponding frame, the message is retrieved and α[k] is applied. When ramp_state is RAMP_DOWN, the acoustic echo cancellation module synchronously increases the nonlinear processing level to mask short-term echo leakage that may occur during gain drop.

[0076] When clipping protection is triggered and the current physical gain value G_current is reduced by a preset attenuation amount in one go, the processor also generates a corresponding gain synchronization message for the reduced current physical gain value G_current. This gain synchronization message may include the current frame sequence number F_seq, the reduced G_current, and the gain amplitude compensation factor α[k] calculated based on G_current. Therefore, even if the gain change is not caused by a regular step update, the acoustic echo cancellation module can still obtain the latest gain change information in a timely manner.

[0077] Step S7: The processor sends a gain synchronization message to the acoustic echo cancellation module, so that when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], it applies the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to that audio frame.

[0078] Specifically, the acoustic echo cancellation module maintains a gain synchronization message queue. After the processor completes a gain write to the physical gain register, it writes the corresponding gain synchronization message to this gain synchronization message queue. The gain synchronization message queue can buffer 8 to 16 gain synchronization messages to cover short-term scheduling delays or audio processing thread delays.

[0079] like Figure 4 As shown, Figure 4 This is a frame-level latching synchronization timing diagram of the gain amplitude compensation factor in the acoustic echo cancellation linkage protection mechanism. It is used to illustrate the timing relationship between three events: the step change of the current physical gain value G_current, the sending of the gain synchronization message, and the frame-by-frame processing of the acoustic echo cancellation module. Figure 4 It can be divided vertically into three lanes, with the three lanes sharing the same time axis. The time axis can be in units of frame sequence number F_seq, with each time frame corresponding to an audio frame, and the frame length can be 10ms to 20ms.

[0080] The first swimlane illustrates the timing of writes to the audio codec's physical gain register. The timeline of the first swimlane is marked with the moment of each G_current step change, and the corresponding G_current value is labeled at the appropriate position, such as 30dB, 28.5dB, 27dB, until G_current reaches the target gain value. Each G_current step change corresponds to one write to the audio codec's physical gain register.

[0081] In the second swimlane, the gain synchronization message queue is shown. Each gain synchronization message corresponds to one G_current change event, and the gain synchronization message includes at least the F_seq value, the G_current value, and the gain amplitude compensation factor α[k]. In terms of timing, the gain synchronization message enters the message queue at the time it is written to the corresponding physical gain register and remains there until the acoustic echo cancellation module consumes the gain synchronization message.

[0082] In lane 3, the frame-by-frame processing timing of the acoustic echo cancellation module is shown. The acoustic echo cancellation module processes the audio frames to be processed in the order of the audio frames, and each audio frame to be processed has a corresponding frame sequence number F_seq. When the F_seq of the audio frame to be processed matches the F_seq in the gain synchronization message, the acoustic echo cancellation module reads α[k] in the gain synchronization message and applies α[k] during the amplitude process of the echo path reference signal of the audio frame to be processed.

[0083] Figure 4 A causal connection can be established between the gain synchronization message and the corresponding audio frame to be processed through message matching. This causal connection means that the acoustic echo cancellation module does not directly apply α[k] according to the message arrival time, but instead matches it according to the frame sequence number F_seq, and then reads and latches α[k] before processing the corresponding audio frame. Therefore, even if there is a short delay in operating system scheduling, audio thread processing, or message queue transmission, F_seq can ensure that the gain amplitude compensation factor applies to the correct audio frame.

[0084] When processing an audio frame, the acoustic echo cancellation module queries the gain synchronization message queue according to the frame sequence number of the audio frame. When the frame sequence number of the audio frame matches F_seq[k] in the gain synchronization message, the acoustic echo cancellation module reads the gain amplitude compensation factor α[k] corresponding to F_seq[k] and applies this gain amplitude compensation factor α[k] during the amplitude update of the echo path reference signal of the audio frame. Before the amplitude update of the echo path reference signal of the audio frame is completed, the gain amplitude compensation factor α[k] is not interrupted or overwritten by another gain synchronization message. In other words, the acoustic echo cancellation module performs gain compensation on a single frame of audio in a non-interruptible, single-operation manner to avoid estimation errors caused by using multiple different compensation factors in the same audio frame.

[0085] In this way, the acoustic echo cancellation module can detect changes in the microphone's physical gain across specific frames, ensuring that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain. Compared to re-estimating the echo path only after gain stabilization, this embodiment reduces residual echo and re-convergence time during gain switching.

[0086] In one embodiment, when the frame sequence number jumps, is lost, or wraps around, the acoustic echo cancellation module recalculates the gain amplitude compensation factor α[k] based on the current physical gain value G_current, and resets or rebuilds the gain synchronization message queue to avoid the old gain synchronization message from being incorrectly applied to subsequent audio frames.

[0087] In a further embodiment, during the step-up update of the current physical gain value G_current towards a lower gain, the acoustic echo cancellation module temporarily increases its nonlinear processing level; once the current physical gain value G_current reaches the target gain value G_target, the nonlinear processing level is restored. This enhances the suppression of short-term residual echoes during gain reduction.

[0088] When clipping protection is triggered and the current physical gain value G_current is reduced in one go, the processor sends the gain synchronization message corresponding to the reduced current physical gain value G_current to the acoustic echo cancellation module. This causes the acoustic echo cancellation module to update the amplitude of the echo path reference signal of the corresponding audio frame according to the gain synchronization message. Thus, the emergency gain reduction process of clipping can also be synchronized with AEC compensation.

[0089] In another embodiment, when the audio operating state switches to two-way call mode, the processor reduces the automatic noise suppression level; when the audio operating state exits two-way call mode, the processor restores the automatic noise suppression level. In one specific embodiment, in two-way call mode, the automatic noise suppression level is reduced from the default level L_ans_default to L_ans_bidir, a reduction of 1 to 2 levels; when exiting two-way call mode, the automatic noise suppression level is restored to L_ans_default. By reducing the automatic noise suppression level in two-way call mode, the probability of near-end speech being falsely suppressed or producing a discontinuous feel can be reduced, improving the naturalness of two-way calls.

[0090] In one application scenario, when a user on the APP presses the intercom button to initiate a one-way intercom, the smart camera receives and decodes the downlink audio stream. In a high-noise environment, the downlink audio noise baseline N_F[k] is relatively high, and the adaptive margin parameter M[k] can be adjusted to 10dB. When the processor detects that the frame energy E_dB[n] of four consecutive decibel domain frames is greater than T_h[k] and SNR[k] is greater than or equal to SNR_th, the state machine switches from silent monitoring mode to one-way intercom mode, and the target gain value is updated to 18dB. During the shadow gain control process, G_current is reduced by 1.5dB every 20ms, and written to the physical gain register at the zero-crossing point, while a gain synchronization message is sent to the acoustic echo cancellation module. After the user releases the intercom button, the timer is suspended until 720ms expires, the state machine switches back to silent monitoring mode, and the target gain value is restored to 30dB.

[0091] In another application scenario, when the smart camera is in two-way communication mode and the speaker volume is high, the microphone signal may be clipped. If the clipping protection logic detects three consecutive clipping frames, the processor immediately reduces the current physical gain value G_current by 6dB and generates a gain synchronization message containing the current frame sequence number F_seq, the reduced G_current, and the corresponding gain amplitude compensation factor α[k]. This causes the acoustic echo cancellation module to adjust the amplitude of the echo path reference signal according to the updated α[k]. When the clipping state is lifted, the processor can resume normal step control, allowing G_current to gradually approach the target gain value corresponding to the current audio operating state.

[0092] In one test case, to verify the technical effect of this embodiment compared to the one-time gain-switching scheme, a comparative test can be conducted on a low-cost IPC camera platform. The test environment can be an echo test environment, and the test results are shown in Table 2.

[0093] Table 2 Comparison of audio effects between this embodiment and the one-time gain jump scheme

[0094] This embodiment also provides an intelligent camera audio parameter adaptive switching system for executing the above-described intelligent camera audio parameter adaptive switching method. The system includes an audio framing module, an energy threshold module, a state discrimination module, a gain mapping module, a shadow gain control module, a register writing module, a gain synchronization module, and an acoustic echo cancellation linkage module.

[0095] The audio framing module is used to acquire the downlink audio stream received by the smart camera, decode and framing the downlink audio stream to obtain multiple downlink audio frames, and assign a frame sequence number F_seq to each downlink audio frame.

[0096] The energy threshold module is used to calculate the decibel-domain frame energy E_dB[n] of each downlink audio frame and determine the dynamic threshold T_h[k] based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k]. In some embodiments, the energy threshold module includes an energy calculation unit and an adaptive threshold calculation unit, wherein the energy calculation unit is used to calculate the RMS energy and the adaptive threshold calculation unit is used to calculate T_h[k] based on N_F[k] and M[k].

[0097] The state discrimination module is used to determine the audio working state of the smart camera based on the comparison result of the decibel domain frame energy E_dB[n] and the dynamic threshold T_h[k], combined with the effective trigger count and the suspend timer, so that the audio working state switches between silent monitoring mode, one-way intercom mode and two-way communication mode. In some embodiments, the state discrimination module includes a state machine unit, which is used to execute the state transition logic between silent monitoring mode, one-way intercom mode and two-way communication mode.

[0098] The gain mapping module is used to determine the target gain value G_target based on the audio operating status.

[0099] The shadow gain control module is used to maintain the target gain value G_target and the current physical gain value G_current at the software layer when the current physical gain value G_current is inconsistent with the target gain value G_target, and to update the current physical gain value G_current in steps according to a preset step size. In some embodiments, the shadow gain control module maintains the target gain value and the current physical gain value, and generates a discrete stepped control sequence.

[0100] The register writing module is used to write the updated physical gain value G_current to the physical gain register of the audio codec at the time corresponding to the zero-crossing point or near-zero-crossing point. The register writing module can include a hardware zero-crossing writing unit and a software zero-crossing writing unit. The hardware zero-crossing writing unit enables hardware zero-crossing detection when the audio codec supports it, and writes to the physical gain register when a zero-crossing point of the audio signal is detected. The software zero-crossing writing unit performs mean elimination processing on the PCM audio data to obtain a DC-bias-free sampling sequence when the audio codec does not support hardware zero-crossing detection. It then finds the zero-crossing point or near-zero-crossing point based on the sampling sequence and writes it to the physical gain register at the time corresponding to the zero-crossing point or near-zero-crossing point.

[0101] The gain synchronization module is used to generate a gain synchronization message after each step update of the current physical gain value G_current. The gain synchronization message includes the frame sequence number F_seq[k] corresponding to this step update, the current physical gain value G_current[k] after the step update, and the gain amplitude compensation factor α[k] calculated based on the current physical gain value G_current[k].

[0102] The acoustic echo cancellation linkage module is used to receive gain synchronization messages, and when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], it applies the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to the audio frame, so that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain.

[0103] The acoustic echo cancellation linkage module may include a message queue unit and a frame matching unit. The message queue unit is used to buffer gain synchronization messages. The frame matching unit is used to query the message queue unit according to the frame sequence number of the audio frame to be processed, and when the frame sequence number of the audio frame to be processed matches F_seq[k] in the gain synchronization message, it reads the gain amplitude compensation factor α[k] corresponding to F_seq[k], and completes the application of the gain amplitude compensation factor α[k] during the amplitude update process of the echo path reference signal of the audio frame to be processed.

[0104] In some embodiments, the system further includes a clipping protection module. The clipping protection module is used to determine whether the number of sampling points reaching or approaching the full-scale amplitude of the analog-to-digital converter in a consecutive preset number of audio frames of the local microphone signal exceeds a preset threshold. If the number exceeds the preset threshold, the module controls the current physical gain value G_current to be reduced by a preset attenuation amount at once, and simultaneously triggers the gain synchronization module to generate a corresponding gain synchronization message.

[0105] In some embodiments, the system further includes an automatic noise suppression linkage module. The automatic noise suppression linkage module is used to reduce the automatic noise suppression level when switching from audio operating mode to two-way communication mode, and to restore the automatic noise suppression level when exiting two-way communication mode from audio operating mode.

[0106] In some embodiments, the system further includes a low signal-to-noise ratio (SNR) suppression module. The low SNR suppression module is used to calculate the signal-to-noise ratio (SNR) SNR[k] based on SNR[k] = E_dB[n] - N_F[k]. When the decibel domain frame energy E_dB[n] of the downlink audio frame is greater than the dynamic threshold T_h[k] but the SNR[k] is lower than the preset SNR threshold SNR_th, the control state discrimination module does not accumulate the effective trigger count and does not use the downlink audio frame to update the downlink audio noise baseline N_F[k].

[0107] The modules described above can be implemented through software programs or through a combination of software and hardware. When implemented through software programs, the program can be stored in memory and executed by the processor to implement the aforementioned adaptive switching method for audio parameters of the smart camera.

[0108] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for adaptive switching of audio parameters in a smart camera, characterized in that, Includes the following steps: Step S1: Obtain the downlink audio stream received by the smart camera, decode and frame the downlink audio stream to obtain multiple downlink audio frames, and assign a frame sequence number F_seq to each downlink audio frame. Based on a unified audio frame clock, associate the frame sequence number F_seq with the speaker playback reference frame and / or the near-end microphone acquisition frame in the corresponding time window. Step S2: Calculate the decibel-domain frame energy E_dB[n] of each downlink audio frame, and determine the dynamic threshold T_h[k] based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k]. Step S3: Based on the comparison result of the decibel domain frame energy E_dB[n] and the dynamic threshold T_h[k], combined with the effective trigger count and the suspension timer, determine the audio working state of the smart camera, and switch the audio working state between silent monitoring mode, one-way intercom mode and two-way call mode. Step S4: Determine the target gain value G_target based on the audio working state, and maintain the target gain value G_target and the current physical gain value G_current in the software layer; Step S5: When the current physical gain value G_current is inconsistent with the target gain value G_target, the current physical gain value G_current is updated step by step according to a preset step size, and at the time corresponding to the zero crossing point or near the zero crossing point, the updated current physical gain value G_current is written into the physical gain register of the audio codec. Step S6: After each step update of the current physical gain value G_current, a gain synchronization message is generated. The gain synchronization message includes the frame sequence number F_seq[k] corresponding to this step update, the current physical gain value G_current[k] after the step update, and the gain amplitude compensation factor α[k] calculated based on the current physical gain value G_current[k]. Step S7: Send the gain synchronization message to the acoustic echo cancellation module, so that when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], it applies the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to that audio frame, so that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain.

2. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S1, the downlink audio stream is decoded and framed, including: The downlink audio stream is decoded to obtain a downlink PCM audio stream; The downlink PCM audio stream is divided into frames with a frame length of 10ms to 20ms to obtain multiple downlink audio frames; Each downlink audio frame is assigned a frame sequence number F_seq using an incremental counting method.

3. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S2, the decibel-domain frame energy E_dB[n] of each downlink audio frame is calculated, and the dynamic threshold T_h[k] is determined based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k], including: The RMS energy E_RMS[n] of the nth downlink audio frame is calculated according to the formula E_RMS[n]=sqrt((1 / L)×Σx_i²), where L is the number of sampling points in the downlink audio frame and x_i is the amplitude of the i-th sampling point in the downlink audio frame; the RMS energy E_RMS[n] is then converted into decibel-domain frame energy E_dB[n]. The downlink audio noise baseline N_F[k] is updated based on the low-energy frames in the decibel domain frame energy E_dB[n] of multiple historical downlink audio frames; The dynamic threshold T_h[k] is determined according to the formula T_h[k]=N_F[k]+M[k]; wherein, N_F[k], M[k] and T_h[k] are all decibel-domain parameters; When the downlink audio noise baseline N_F[k] is higher than the high noise environment judgment threshold T_noisy, the adaptive margin parameter M[k] is adjusted towards the upper limit M_max. When the downlink audio noise baseline N_F[k] is lower than the quiet environment judgment threshold T_quiet, the adaptive margin parameter M[k] is adjusted in the direction of the lower limit M_min; The downlink audio noise baseline N_F[k] is taken as the moving average of low-energy frames in the most recent 80 to 120 frames; the upper limit M_max of the adaptive margin parameter M[k] is 10dB, the lower limit M_min is 6dB, and the change of the adaptive margin parameter M[k] every 100ms does not exceed 1dB.

4. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S3, determining the audio operating status of the smart camera includes: When the decibel frame energy E_dB[n] of N consecutive downlink audio frames is greater than the dynamic threshold T_h[k], a valid downlink voice signal is determined to exist; when the decibel frame energy E_dB[n] of the downlink audio frame is lower than the dynamic threshold T_h[k], the suspension timer is started; when the suspension timer reaches the preset suspension time T_hangover, the valid downlink voice signal is determined to have disappeared; when the decibel frame energy E_dB[n] of the downlink audio frame is greater than the dynamic threshold T_h[k] again, the suspension timer is reset; where N is 3 to 5, and T_hangover is 5. 00ms~1s; Based on the decibel-domain frame energy E_dB[n] and the downlink audio noise baseline N_F[k], the signal-to-noise ratio SNR[k] is calculated according to SNR[k]=E_dB[n]-N_F[k]; When the decibel-domain frame energy E_dB[n] is greater than the dynamic threshold T_h[k] and the signal-to-noise ratio SNR[k] is lower than the preset signal-to-noise ratio threshold SNR_th, the effective trigger count is not accumulated, and the downlink audio frame is not used to update the downlink audio noise baseline N_F[k]; The downlink audio noise baseline N_F[k] is updated according to the low-energy frame.

5. The intelligent camera audio parameter adaptive switching method according to claim 4, characterized in that, In step S3, the audio working state is switched between silent monitoring mode, one-way intercom mode, and two-way communication mode, including: In silent monitoring mode, when a valid downlink voice signal is detected, switch to one-way intercom mode; In one-way intercom mode, when continuous local voice activity is detected, switch to two-way call mode; In two-way call mode, when it is determined that the local continuous voice activity has disappeared and the downlink valid voice signal still exists, switch to one-way intercom mode. In one-way intercom mode or two-way call mode, when it is determined that the downlink valid voice signal has disappeared, switch to silent monitoring mode; The determination of the existence of local continuous speech activity includes: calculating the local RMS energy E_local[n] of the near-end audio frames captured by the local microphone; comparing the local RMS energy E_local[n] with the local speech detection threshold T_local; determining the existence of local continuous speech activity when the local RMS energy E_local[n] of N consecutive near-end audio frames is greater than the local speech detection threshold T_local; wherein, N_local is 2 to 4.

6. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S4, the target gain value G_target is determined based on the audio operating state, including: In silent monitoring mode, the target gain value G_target is determined as the first gain value; In one-way intercom mode, the target gain value G_target is determined as the second gain value; In two-way communication mode, the target gain value G_target is determined as the third gain value; Wherein, the first gain value is greater than the second gain value, and the second gain value is greater than the third gain value.

7. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S5, the current physical gain value G_current is updated in steps according to a preset step size, and at the time corresponding to the zero-crossing point or near-zero-crossing point, the updated current physical gain value G_current is written into the physical gain register of the audio codec. Specifically, this includes: The current physical gain value G_current is updated by incrementing or decrementing in steps no greater than 1.5dB. A step update is performed every 10ms to 20ms until the current physical gain value G_current reaches the target gain value G_target. When the audio codec supports hardware zero-crossing detection, the hardware zero-crossing detection function is enabled, and when the zero-crossing point of the audio signal is detected, the current physical gain value G_current after step update is written to the physical gain register. When the audio codec does not support hardware zero-crossing detection, software zero-crossing detection is performed on the PCM audio data, and when the software detects a zero-crossing point or near-zero-crossing point, the updated physical gain value G_current is written to the physical gain register. The software zero-crossing detection of PCM audio data includes: calculating the sampling mean μ of the current audio frame; subtracting the sampling mean μ from the amplitude of each sampling point in the current audio frame to obtain the sampling sequence x'[i] after DC bias removal; determining that there is a zero-crossing point between x'[i] and x'[i+1] when x'[i]×x'[i+1]≤0; and determining that there is an approximate zero-crossing point when no zero-crossing point is detected, and the sampling time that satisfies |x'[i]|<ε; where ε is 0.5% to 1% of the full-scale amplitude of the analog-to-digital converter. In step S5, clipping status is also detected for the local microphone signal. The detection of clipping status includes: determining whether the number of sampling points that reach or approach the full-scale amplitude of the analog-to-digital converter in a consecutive preset number of audio frames exceeds a preset number threshold; when the preset number threshold is exceeded, clipping is determined to have occurred, the step update is skipped, and the current physical gain value G_current is reduced by a preset attenuation amount in one go.

8. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S6, the gain amplitude compensation factor α[k] is calculated according to the following formula: α[k]=10^(G_current[k] / 20)×α_base; Where G_current[k] is the current physical gain value after the k-th step update, and α_base is the base compensation coefficient; The gain synchronization message also includes a step direction identifier and a timestamp; the step direction identifier is used to indicate whether the current physical gain value G_current is in a gain reduction phase, a gain increase phase, or a stable phase; the timestamp is used to reset or rebuild the gain synchronization message queue when the frame sequence number changes. When the current physical gain value G_current is reduced by a preset attenuation amount in one go, in step S6, a corresponding gain synchronization message is generated for the current physical gain value G_current after the one-time reduction.

9. The method for adaptive switching of audio parameters of an intelligent camera according to claim 1, characterized in that, In step S7, the gain synchronization message is sent to the acoustic echo cancellation module, so that when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], it applies the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to that audio frame. Specifically, this includes: S71. Write the gain synchronization message into the gain synchronization message queue; S72. The acoustic echo cancellation module queries the gain synchronization message queue according to the frame sequence number of the audio frame to be processed; S73. When the frame sequence number of the audio frame to be processed is consistent with F_seq[k] in the gain synchronization message, read the gain amplitude compensation factor α[k] corresponding to F_seq[k]. S74. Before processing the audio frame to be processed, read and latch the gain amplitude compensation factor α[k] corresponding to the audio frame to be processed, keep the gain amplitude compensation factor α[k] unchanged before the processing of the audio frame to be processed ends, and apply it to the echo path reference signal corresponding to the audio frame to be processed to adjust the amplitude of the echo path reference signal. When the current physical gain value G_current is reduced by a preset attenuation amount at once, in step S7, the gain synchronization message corresponding to the reduced current physical gain value G_current is sent to the acoustic echo cancellation module. During the step update of the current physical gain value G_current towards a lower gain direction, the nonlinear processing level in the acoustic echo cancellation module is increased; when the current physical gain value G_current reaches the target gain value G_target, the nonlinear processing level is restored.

10. A smart camera audio parameter adaptive switching system, characterized in that, include: The audio framing module is used to acquire the downlink audio stream received by the smart camera, decode and framing the downlink audio stream to obtain multiple downlink audio frames, and assign a frame sequence number F_seq to each downlink audio frame. Based on a unified audio frame clock, the frame sequence number F_seq is associated with the speaker playback reference frame and / or the near-end microphone acquisition frame within the corresponding time window. The energy threshold module is used to calculate the decibel-domain frame energy E_dB[n] of each downlink audio frame and determine the dynamic threshold T_h[k] based on the downlink audio noise baseline N_F[k] and the adaptive margin parameter M[k]. The state discrimination module is used to determine the audio working state of the smart camera based on the comparison result of the decibel domain frame energy E_dB[n] and the dynamic threshold T_h[k], combined with the effective trigger count and the suspension timer, so that the audio working state can be switched between silent monitoring mode, one-way intercom mode and two-way call mode. The gain mapping module is used to determine the target gain value G_target based on the audio operating state; The shadow gain control module is used to maintain the target gain value G_target and the current physical gain value G_current in the software layer, and when the current physical gain value G_current is inconsistent with the target gain value G_target, to update the current physical gain value G_current in steps according to a preset step size; The register writing module is used to write the updated physical gain value G_current to the physical gain register of the audio codec at the moment corresponding to the zero crossing or near zero crossing. The gain synchronization module is used to generate a gain synchronization message after each step update of the current physical gain value G_current. The gain synchronization message includes the frame sequence number F_seq[k] corresponding to this step update, the current physical gain value G_current[k] after the step update, and the gain amplitude compensation factor α[k] calculated based on the current physical gain value G_current[k]. The acoustic echo cancellation linkage module is used to receive the gain synchronization message and, when the acoustic echo cancellation module processes an audio frame that is in the same frame clock window as F_seq[k], apply the gain amplitude compensation factor α[k] to the amplitude of the echo path reference signal corresponding to the audio frame so that the echo path reference signal and the actual microphone pickup signal remain linearly aligned in the amplitude domain. The register writing module includes a hardware zero-crossing writing unit and a software zero-crossing writing unit. The hardware zero-crossing writing unit is used to enable the hardware zero-crossing detection function when the audio codec supports it, and write to the physical gain register when a zero-crossing point of the audio signal is detected. The software zero-crossing writing unit is used to perform mean elimination processing on the PCM audio data to obtain a sampled sequence after DC bias removal when the audio codec does not support the hardware zero-crossing detection function, and find the zero-crossing point or near-zero-crossing point according to the sampled sequence, and write to the physical gain register at the time corresponding to the zero-crossing point or the near-zero-crossing point. The acoustic echo cancellation linkage module includes a message queue unit and a frame matching unit. The message queue unit is used to cache the gain synchronization message. The frame matching unit is used to query the message queue unit according to the frame sequence number of the audio frame to be processed, and when the frame sequence number of the audio frame to be processed is consistent with F_seq[k] in the gain synchronization message, it reads the gain amplitude compensation factor α[k] corresponding to F_seq[k], and applies the gain amplitude compensation factor α[k] during the amplitude update process of the echo path reference signal of the audio frame to be processed.

Images