Face detection real-time synchronous pickup data processing method and system

Abstract

The invention relates to the technical field of image processing, and discloses a real-time synchronous pickup data processing method and system for face detection, and the method comprises the steps: carrying out the face detection processing of collected video stream data, starting timing when a face which accords with a preset posture feature is detected in a preset distance range, and carrying out the timing; when the continuous detection time reaches a preset threshold value, an effective face detection signal is generated; in response to the effective face detection signal, when the signal duration reaches a preset threshold value, automatically triggering a pickup device to start recording and synchronously controlling the state of an indicator lamp, and generating original audio data; selecting a beam forming scheme, and performing adaptive beam forming processing on the original audio data to obtain mode specific audio data; performing multi-stage audio enhancement processing on the mode specific audio data to obtain enhanced audio data, and performing role structured transcription on the enhanced audio data to obtain voiceprint identification text data; according to the invention, the efficiency of face detection real-time synchronous pickup data processing can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for real-time synchronous audio data processing for face detection. Background Technology

[0002] Existing technologies have significant shortcomings in the coordination of face detection and audio triggering. Face detection in video stream data does not combine preset distance ranges and pose features for multi-dimensional verification. It only judges the detection results through simple face contour recognition, which easily misidentifies faces that are not at the target distance or do not meet the pose requirements as valid detection objects, resulting in false triggering of the audio pickup device. At the same time, the detection signal is not timed for duration and the consistency between consecutive frames is not verified, and it is impossible to filter out instantaneous interference or unstable detection signals, resulting in low reliability of the generated valid face detection signals. When responding to the detection signal, confidence assessment and graded triggering are not carried out. Only a single threshold is used to trigger the audio pickup device, which cannot dynamically adjust the response strategy according to the confidence of the signal. It is easy to have improper audio pickup timing or missed triggering, which affects the effectiveness and timeliness of the original audio data acquisition.

[0003] Existing technologies have significant shortcomings in audio data processing and transcription. They fail to select an appropriate beamforming scheme based on the configured recording mode for the raw audio data, employing only fixed beam processing or omnidirectional recording. This fails to eliminate channel phase deviations and suppress residual noise, making it impossible to accurately focus on the target sound source. The generated audio data is easily mixed with environmental noise and interference signals, resulting in poor sound quality. Audio enhancement processing does not eliminate echoes, suppress noise, and improve speech clarity in stages; it only performs single noise reduction, making it difficult to comprehensively improve audio quality. The enhanced audio data has insufficient signal-to-noise ratio and clarity. During character-based structured transcription, voiceprint feature vectors are not extracted for clustering and grouping; the audio is simply converted to text, failing to associate speaker identity with corresponding text. The generated transcribed text lacks character identification and cannot meet the needs of accurate audio data parsing in multi-speaker scenarios. Summary of the Invention

[0004] This invention provides a method and system for real-time synchronous audio data processing for face detection, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for real-time synchronous audio data processing for face detection, comprising: S1. Perform face detection processing on the acquired video stream data. When a face matching the preset pose features is detected within a preset distance range, start timing. When the continuous detection time reaches a preset threshold, generate an effective face detection signal for the video stream data. S2. In response to the valid face detection signal, when the signal duration reaches the preset threshold, the audio pickup device is automatically triggered to start recording and the indicator light status is controlled synchronously to generate the original audio data of the video stream data. S3. Select the corresponding beamforming scheme according to the preset configuration of the radio reception mode, and perform adaptive beamforming processing on the original audio data to obtain the mode-specific audio data of the original audio data. S4. Perform multi-level audio enhancement processing on the mode-specific audio data to obtain enhanced audio data of the mode-specific audio data; S5. The enhanced audio data is transmitted to a remote server, and the enhanced audio data is transcribed into a role-based structure to obtain the voiceprint identifier text data of the enhanced audio data.

[0006] In a preferred embodiment, the face detection processing of the acquired video stream data includes starting a timer when a face matching a preset pose feature is detected within a preset distance range, and generating a valid face detection signal for the video stream data when the continuous detection time reaches a preset threshold. Multi-pose facial feature analysis is performed on the video stream data to obtain key feature information of the video stream data; The matching degree between the key feature information and the preset posture features is analyzed, and a timer is started when the matching degree continuously reaches a preset threshold. The key feature information is continuously verified for consistency across frames to ensure the stability of the attitude features; When the consecutive inter-frame consistency verification passes and the duration reaches the preset threshold, a valid face detection signal is generated for the video stream data.

[0007] In a preferred embodiment, in response to the valid face detection signal, when the signal duration reaches the preset threshold, the audio pickup device is automatically triggered to start recording and the indicator light status is simultaneously controlled to generate the raw audio data of the video stream data, including: The confidence factor of the effective face detection signal is obtained by evaluating the confidence level of the effective face detection signal. Based on the quantization comparison between the confidence factor and the preset threshold, the confidence factor is mapped to the corresponding confidence level, and a graded triggering instruction for the video stream data is generated. Based on the hierarchical triggering command, the video stream data is subjected to multi-level response to obtain hierarchical response video data of the video stream data; Based on the graded response video data, the raw audio data of the video stream data is generated.

[0008] In a preferred embodiment, the confidence factor is calculated using the following formula: ; In the formula, To determine the overall confidence factor, For time weighting coefficients, For signal duration, For the preset time threshold, This is the intensity weighting coefficient. The current signal strength, For maximum signal strength, This represents the historical confidence level average. This is the previous confidence level value. The fluctuation weighting coefficient is used. The variance of the signal variation This represents the absolute value of the feature difference between the historical confidence mean and the previous confidence value. These are the continuous weighting coefficients.

[0009] In a preferred embodiment, the step of selecting a corresponding beamforming scheme according to a preset configured reception mode and performing adaptive beamforming processing on the original audio data to obtain mode-specific audio data of the original audio data includes: Multi-channel spatial characteristic analysis is performed on the original audio data to obtain the spatial covariance matrix of the original audio data; Based on the configured radio reception mode, beamforming optimization is performed on the spatial covariance matrix to obtain the acoustic steering vector set of the original audio data; Calculate the beamforming weight vector of the original audio data, wherein the formula for calculating the beamforming weight vector is: ; In the formula, A weight vector is formed for the beam. Let be the spatial covariance matrix. For acoustic steering vector set, This is the conjugate transpose operator. The regularization coefficient is . It is the identity matrix. The matrix inversion operator; Based on the beamforming weight vector, the original audio data is subjected to spatial filtering to obtain mode-specific audio data of the original audio data.

[0010] In a preferred embodiment, the step of performing spatial filtering on the original audio data based on the beamforming weight vector to obtain mode-specific audio data of the original audio data includes: Eliminate the phase deviation of the channel audio signals in the audio acquisition device to obtain the coherent multi-channel signal of the original audio data; Based on the preset target pickup direction information, determine the spatial response characteristics of the microphone array; Generate corresponding beamforming weighting coefficients based on the spatial response characteristics; Based on the beamforming weighting coefficients, the coherent multichannel signal is spatially filtered to obtain the directional pickup beam of the coherent multichannel signal; Suppressing residual noise from the directional pickup beam generates pattern-specific audio data from the original audio data.

[0011] In a preferred embodiment, eliminating the phase deviation of the channel audio signals in the audio acquisition device to obtain the coherent multi-channel signal of the original audio data includes: Acquire the channel audio signals of the microphone array in the audio acquisition device; Perform inter-channel phase consistency analysis on the channel audio signal to obtain the phase deviation characteristics of the channel audio signal; Based on the phase deviation characteristics, the phase deviation between the channel audio signals is eliminated to obtain the coherent multi-channel signal of the channel audio signal.

[0012] In a preferred embodiment, the step of performing multi-level audio enhancement processing on the pattern-specific audio data to obtain enhanced audio data of the pattern-specific audio data includes: Eliminate the echo component of the mode-specific audio data to generate primary processed audio data of the mode-specific audio data; The primary processed audio data is subjected to noise suppression processing to obtain secondary processed audio data of the mode-specific audio data; The signal-to-noise ratio and clarity of the speech segments in the secondary processed audio data are improved to obtain enhanced audio data of the mode-specific audio data.

[0013] In a preferred embodiment, the step of transmitting the enhanced audio data to a remote server and performing role-based structured transcription on the enhanced audio data to obtain the voiceprint identifier text data of the enhanced audio data includes: The enhanced audio data is transmitted to the data receiving end of the remote server, and a secure transmission confirmation signal for the enhanced audio data is generated. In response to the secure transmission confirmation signal, voice activity detection is performed on the enhanced audio data to obtain the time stamp information of the enhanced audio data; Based on the time stamp information, extract the voiceprint feature vector of the detected speech segment; The voiceprint feature vectors are clustered and grouped according to the speech segments of different speakers to obtain the role-separated audio data of the voiceprint feature vectors; The character-separated audio data is converted into text data to obtain the original transcribed text of the character-separated audio data; Based on the voiceprint feature vector and the time stamp information, the original transcribed text is associated and fused with the corresponding speaker identity to obtain the voiceprint identifier text data of the enhanced audio data.

[0014] To address the aforementioned problems, the present invention also provides a real-time synchronous audio data processing system for face detection, the system comprising: The effective face detection module is used to perform face detection processing on the acquired video stream data. When a face that matches the preset pose features is detected within a preset distance range, a timer is started, and when the continuous detection time reaches a preset threshold, an effective face detection signal of the video stream data is generated. The intelligent recording trigger module is used to respond to the valid face detection signal. When the signal duration reaches the preset threshold, it automatically triggers the sound pickup device to start recording and synchronously controls the indicator light status to generate the original audio data of the video stream data. An adaptive beamforming module is used to select a corresponding beamforming scheme according to a preset configured radio reception mode, and to perform adaptive beamforming processing on the original audio data to obtain mode-specific audio data of the original audio data. A multi-level audio enhancement module is used to perform multi-level audio enhancement processing on the mode-specific audio data to obtain enhanced audio data of the mode-specific audio data. The character structure transcription module is used to transmit the enhanced audio data to a remote server and perform character structure transcription on the enhanced audio data to obtain the voiceprint identifier text data of the enhanced audio data.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention significantly improves the effectiveness and reliability of face detection and sound pickup coordination through precise face detection and intelligent sound pickup triggering. Multi-pose face feature analysis is performed on the acquired video stream data, analyzing the matching degree between key feature information and preset pose features. Combined with continuous inter-frame consistency verification and duration threshold judgment, a highly reliable effective face detection signal is generated, ensuring that the detection results can accurately identify the target face. The confidence level of the effective face detection signal is evaluated and mapped to the corresponding confidence level, generating hierarchical trigger commands to dynamically adjust the sound pickup response strategy, synchronously controlling the start-up of the sound pickup device and the status of indicator lights, ensuring accurate timing and stable process of raw audio data acquisition, and providing a high-quality data foundation for subsequent audio processing.

[0016] 2. This invention significantly improves audio data quality and transcription practicality through scientific audio processing and structured transcription. Based on a preset recording mode, an appropriate beamforming scheme is selected. Through multi-channel spatial characteristic analysis, beamforming optimization, and spatial filtering, channel phase deviations are eliminated and residual noise is suppressed, generating pattern-specific audio data with strong directionality and low noise. The pattern-specific audio data undergoes multi-level enhancement processing, including echo cancellation, noise suppression, and speech intelligibility improvement, further optimizing audio quality. After the enhanced audio data is transmitted to a remote server, voice activity detection, voiceprint feature vector extraction, and clustering are used to separate audio from different speakers. Then, voiceprint features and time stamp information are correlated to complete role-based structured transcription, generating text data with voiceprint identification, fully meeting the needs for accurate audio data analysis and efficient utilization in various scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for real-time synchronous audio data processing for face detection according to an embodiment of the present invention. Figure 2 This is a functional block diagram of a face detection real-time synchronous audio data processing system according to an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for real-time synchronous audio data processing for face detection. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] Reference Figure 1The diagram shown is a flowchart illustrating a real-time synchronous audio data processing method for face detection according to an embodiment of the present invention. In this embodiment, the real-time synchronous audio data processing method for face detection includes: S1. Perform face detection processing on the acquired video stream data. When a face matching the preset pose features is detected within a preset distance range, start timing. When the continuous detection time reaches a preset threshold, generate an effective face detection signal for the video stream data. In this embodiment of the invention, the face detection processing of the acquired video stream data, which involves starting a timer when a face matching a preset pose feature is detected within a preset distance range, and generating a valid face detection signal for the video stream data when the continuous detection time reaches a preset threshold, includes: Multi-pose facial feature analysis is performed on the video stream data to obtain key feature information of the video stream data; The matching degree between the key feature information and the preset posture features is analyzed, and a timer is started when the matching degree continuously reaches a preset threshold. The key feature information is continuously verified for consistency across frames to ensure the stability of the attitude features; When the consecutive inter-frame consistency verification passes and the duration reaches the preset threshold, a valid face detection signal is generated for the video stream data.

[0021] Specifically, the video stream data is analyzed frame by frame, and features such as facial contours, facial angles, eye-opening status, and head tilt are extracted from each frame. These features are then integrated into key feature information. Among them, multi-pose facial feature analysis covers facial feature extraction under different poses such as frontal, side, head-up, and head-down, ensuring that the key feature information can comprehensively reflect the various poses of the face in the video stream.

[0022] Furthermore, the matching degree of facial contours, facial angle deviation, eye-opening state conformity, and head tilt degree adaptability in the key feature information are compared with the corresponding standard values ​​in the preset posture features to calculate the overall matching degree. When the matching degree remains at or above the preset threshold for multiple consecutive frames, the timing function is immediately started to record the duration.

[0023] Furthermore, the key feature information of multiple consecutive frames is compared to check whether the positional changes of facial contours, the fluctuation range of facial angles, the duration of the open-eye state, and the stability of head tilt are within the set reasonable range between adjacent frames. If the changes of key feature information in all consecutive frames do not exceed the range, the consistency verification between consecutive frames is deemed to have passed, and the posture features are confirmed to be in a stable state.

[0024] Furthermore, when the state of continuous inter-frame consistency verification is maintained and the duration recorded by the timing function reaches a preset threshold, the system automatically generates a valid face detection signal for the video stream data. This signal includes the key facial feature information detected, the continuous detection time, and the verification pass mark, so as to confirm that there is a valid face in the video stream that meets the requirements.

[0025] In summary, multi-pose facial feature analysis extracts key feature information, covering common poses in various business scenarios such as frontal, side, head-down, and wearing masks. Combined with preset distance and angle ranges, it ensures that faces that meet the requirements can be recognized, providing comprehensive feature support for matching and verification.

[0026] In summary, the system analyzes the matching degree between key features and preset poses, starts timing only when the target is consistently met, filters out momentary non-target faces or pose mismatches, avoids the sound pickup device from starting due to false detection, and ensures accurate detection signals.

[0027] In summary, the continuous inter-frame consistency verification checks the magnitude of facial feature changes, passing only those changes within a reasonable range and with stable posture, excluding instability caused by shaking or occlusion, thus further improving detection reliability. A valid signal is only generated after successful verification and a specified duration, ensuring the detection of a stable target face and providing a reliable prerequisite for subsequent accurate recording by the audio pickup device and automatic segmentation of recordings.

[0028] S2. In response to the valid face detection signal, when the signal duration reaches the preset threshold, the audio pickup device is automatically triggered to start recording and the indicator light status is controlled synchronously to generate the original audio data of the video stream data. In this embodiment of the invention, in response to the valid face detection signal, when the signal duration reaches the preset threshold, the audio pickup device is automatically triggered to start recording and the indicator light status is controlled synchronously to generate the raw audio data of the video stream data, including: The confidence factor of the effective face detection signal is obtained by evaluating the confidence level of the effective face detection signal. Based on the quantization comparison between the confidence factor and the preset threshold, the confidence factor is mapped to the corresponding confidence level, and a graded triggering instruction for the video stream data is generated. Based on the hierarchical triggering command, the video stream data is subjected to multi-level response to obtain hierarchical response video data of the video stream data; Based on the graded response video data, the raw audio data of the video stream data is generated.

[0029] The confidence factor is calculated using the following formula: ; In the formula, To determine the overall confidence factor, For time weighting coefficients, For signal duration, For the preset time threshold, This is the intensity weighting coefficient. The current signal strength, For maximum signal strength, This represents the historical confidence level average. This is the previous confidence level value. The fluctuation weighting coefficient is used. The variance of the signal variation This represents the absolute value of the feature difference between the historical confidence mean and the previous confidence value. These are the continuous weighting coefficients.

[0030] Specifically, the key facial feature information, continuous detection time, and verification pass flag contained in the effective face detection signal are analyzed to evaluate the reliability of the signal. Specifically, the completeness of the key feature information, the difference between the continuous detection time and the preset threshold, and the validity of the verification pass flag are checked. These evaluation results are integrated into a confidence factor to quantitatively represent the credibility of the effective face detection signal.

[0031] Furthermore, the confidence factor is numerically compared with a preset threshold to determine the range in which the confidence factor falls. Each range corresponds to a specific confidence level. The confidence factor is converted into the corresponding confidence level based on the range it falls in, and then a graded trigger command is generated based on the confidence level. This command clarifies the response level of the subsequent pickup device and indicator light.

[0032] Furthermore, based on the response level specified in the hierarchical trigger command, the audio pickup device and indicator light corresponding to the video stream data are controlled. If the hierarchical trigger command is high-level, the audio pickup device is controlled to start recording with high sensitivity, and the indicator light flashes at a high frequency. If it is medium-level, the audio pickup device starts recording with medium sensitivity, and the indicator light flashes at a medium frequency. If it is low-level, the audio pickup device starts recording with basic sensitivity, and the indicator light flashes at a low frequency. Through this multi-level response processing, hierarchical response video data is obtained.

[0033] Furthermore, sound information recorded by the audio pickup device at the corresponding response level is extracted from the graded response video data. This sound information includes the ambient sound and related audio content corresponding to the video stream data acquisition process. The format and integrity of this sound information are checked to ensure that there are no missing or interference issues, and finally, the original audio data of the video stream data is formed.

[0034] Specifically, the time weight coefficient is determined based on the degree of influence of the time factor on the overall confidence in the effective face detection signal. In the system's preset weight configuration system, a fixed weight ratio is assigned to the time dimension based on the correlation analysis between signal duration and detection accuracy in historical data. This ratio is the time weight coefficient.

[0035] Furthermore, the signal duration is derived from the timing record of the effective face detection signal. The duration is the time from when the timing function is started until the timing stops when the confidence factor is evaluated. The preset time threshold is a standard time length set in advance by the system to determine whether the effective face detection signal duration meets the standard. This time length is determined based on test data from a large number of face detection scenarios and is stored in the system's parameter configuration module.

[0036] Furthermore, the intensity weighting coefficient is determined based on the influence of the intensity of the valid face detection signal on the overall confidence level. By analyzing the reliability of the detection results under different signal intensities, a fixed proportion is allocated to the intensity dimension in the system weighting configuration system; this proportion is the intensity weighting coefficient. The current signal strength is obtained by detecting the signal amplitude of the valid face detection signal during transmission and processing. The signal detection module in the system collects the signal amplitude in real time and obtains the current signal strength after standardization.

[0037] Furthermore, the maximum signal strength is determined during the system's design and debugging phases by testing the highest signal amplitude achievable by valid face detection signals. This value is stored in the system as a benchmark to measure the relative level of the current signal strength. The historical confidence average is obtained by statistically calculating the confidence factors of all previously generated valid face detection signals. The result of summing all historical confidence factors and dividing by the total number of confidence factors is the historical confidence average.

[0038] Furthermore, the previous confidence value is the confidence factor generated when the system last evaluated the confidence of valid face detection signals. This value is stored in the system's historical data recording module and can be directly retrieved and used. The continuity weight coefficient is determined based on the degree of influence of the continuity factor of valid face detection signals on the overall confidence. Combining the correlation between signal continuity and detection stability in historical data, a fixed proportion is allocated to the continuity dimension in the system's weight configuration system. This proportion is the continuity weight coefficient.

[0039] Furthermore, the fluctuation weight coefficient is determined based on the impact of fluctuation factors in the effective face detection signal on the overall confidence level. By analyzing the relationship between signal fluctuation amplitude and detection reliability, a fixed proportion is allocated to the fluctuation dimension in the system weight configuration system; this proportion is the fluctuation weight coefficient. The signal variation variance is calculated from multiple signal intensity samples of the effective face detection signal over a duration period. First, the average of these samples is calculated; then, the squared differences between each sample and the average are summed, and finally, the result is divided by the number of samples; the result is the signal variation variance.

[0040] Furthermore, the significance of this formula lies in its ability to accurately calculate the comprehensive confidence factor by integrating the characteristics of effective face detection signals across four dimensions: time, intensity, continuity, and fluctuation, thus comprehensively reflecting the reliability of effective face detection signals. The calculation involves first multiplying the time weight coefficient by the square of the ratio of the signal duration to a preset time threshold to obtain the contribution of the time dimension to the confidence factor; then multiplying the intensity weight coefficient by the result calculated using the natural logarithm function, which takes the sum of the current signal intensity and the maximum signal intensity plus one as parameters, to obtain the contribution of the intensity dimension; next, multiplying the continuity weight coefficient by one and subtracting the absolute value of the difference between the historical confidence mean and the previous confidence value, and the ratio of the historical confidence mean to the historical confidence factor, to obtain the contribution of the continuity dimension; then multiplying the fluctuation weight coefficient by the signal variation variance to obtain the contribution of the fluctuation dimension; finally, summing the contributions of the four dimensions yields the comprehensive confidence factor. This calculation method considers the feature differences of each dimension of the signal and highlights the influence of different dimensions on the confidence level through weighting coefficients, ensuring that the comprehensive confidence factor can objectively and accurately measure the credibility of effective face detection signals and provide a reliable basis for the generation of subsequent hierarchical triggering instructions.

[0041] In summary, a confidence factor is generated by evaluating the confidence of valid face detection signals. The confidence level of the signal is quantified by comprehensively considering dimensions such as signal duration, intensity, and stability. This avoids false responses to low-confidence signals due to judgment based on a single dimension, and provides a reliable basis for subsequent triggering.

[0042] In summary, by comparing the confidence factor with a preset threshold to map the confidence level, a tiered trigger command is generated. Different response strategies can be matched according to different requirements for signal credibility in different business scenarios, avoiding the limitations of a single trigger mode.

[0043] In summary, the system provides multi-level responses to the video stream based on hierarchical triggering commands. For example, when the confidence level is high, the audio pickup device records with high sensitivity and the indicator light flashes at high frequency; when the confidence level is low, the audio pickup device records with basic sensitivity and flashes at low frequency. This allows for flexible adaptation between the audio pickup device and the indicator light status, balancing recording quality and energy consumption.

[0044] In summary, by combining graded response video data to generate raw audio data, the system ensures accurate timing of recording start-up and stable process, reduces invalid audio introduced by improper triggering, and provides a high-quality data foundation for subsequent beamforming, audio enhancement, and other processing, thus meeting the business system's demand for high-quality audio sources.

[0045] In summary, the formula integrates four core dimensions: signal duration, current intensity, historical continuity, and fluctuation. By assigning reasonable influence proportions to each dimension through time weighting coefficients and intensity weighting coefficients, it avoids the one-sidedness of single-dimensional evaluation and can comprehensively reflect the authenticity and reliability of effective face detection signals, providing accurate quantitative basis for subsequent triggering decisions.

[0046] In general, the weighting coefficients can be adjusted according to the signal characteristics requirements of different business scenarios. For example, the business hall scenario has high requirements for signal stability, so the time weighting coefficient and the continuity weighting coefficient can be increased. Furthermore, parameters such as the preset time threshold and maximum signal strength can be set in combination with the actual detection range and equipment performance of the scenario, so that the confidence factor calculation can be adapted to diverse application scenarios.

[0047] In summary, by introducing the difference between the historical confidence mean and the previous confidence value, and combining it with the signal variation variance, the continuous stability of the signal can be effectively considered, and the interference of transient abnormal signals on the confidence assessment can be suppressed. At the same time, the natural logarithm function can smooth the assessment bias caused by drastic changes in intensity, ensuring that the confidence factor output is stable and reliable.

[0048] In summary, the comprehensive confidence factor calculated by the formula can be accurately mapped to different confidence levels, thereby generating appropriate hierarchical triggering instructions. This makes the status control of the pickup device and indicator lights more consistent with the actual credibility of the signal, avoiding improper pickup timing or waste of resources due to ambiguous evaluation, ensuring the quality of raw audio data acquisition, and meeting the business system's demand for high-quality audio sources.

[0049] S3. Select the corresponding beamforming scheme according to the preset configuration of the audio reception mode, and perform adaptive beamforming processing on the original audio data to obtain the mode-specific audio data of the original audio data. In this embodiment of the invention, the step of selecting a corresponding beamforming scheme according to a preset configured reception mode and performing adaptive beamforming processing on the original audio data to obtain mode-specific audio data of the original audio data includes: Multi-channel spatial characteristic analysis is performed on the original audio data to obtain the spatial covariance matrix of the original audio data; Based on the configured radio reception mode, beamforming optimization is performed on the spatial covariance matrix to obtain the acoustic steering vector set of the original audio data; Calculate the beamforming weight vector of the original audio data, wherein the formula for calculating the beamforming weight vector is: ; In the formula, A weight vector is formed for the beam. Let be the spatial covariance matrix. For acoustic steering vector set, This is the conjugate transpose operator. The regularization coefficient is . It is the identity matrix. The matrix inversion operator; Based on the beamforming weight vector, the original audio data is subjected to spatial filtering to obtain mode-specific audio data of the original audio data.

[0050] The process of performing spatial filtering on the original audio data based on the beamforming weight vector to obtain mode-specific audio data of the original audio data includes: Eliminate the phase deviation of the channel audio signals in the audio acquisition device to obtain the coherent multi-channel signal of the original audio data; Based on the preset target pickup direction information, determine the spatial response characteristics of the microphone array; Generate corresponding beamforming weighting coefficients based on the spatial response characteristics; Based on the beamforming weighting coefficients, the coherent multichannel signal is spatially filtered to obtain the directional pickup beam of the coherent multichannel signal; Suppressing residual noise from the directional pickup beam generates pattern-specific audio data from the original audio data.

[0051] The process of eliminating phase deviations in the channel audio signals of the audio acquisition device to obtain coherent multi-channel signals of the original audio data includes: Acquire the channel audio signals of the microphone array in the audio acquisition device; Perform inter-channel phase consistency analysis on the channel audio signal to obtain the phase deviation characteristics of the channel audio signal; Based on the phase deviation characteristics, the phase deviation between the channel audio signals is eliminated to obtain the coherent multi-channel signal of the channel audio signal.

[0052] Specifically, the raw audio data is analyzed channel by channel to extract information on the propagation direction, phase difference, and amplitude variation of the audio signal in each channel. This information encompasses the spatial distribution characteristics of the raw audio data. By analyzing the correlation between audio signals in different channels, the degree of mutual influence between the signals of each channel at different time points is calculated. These analysis results are then organized into a matrix, where each element corresponds to the degree of correlation between two channel signals, ultimately forming the spatial covariance matrix of the raw audio data.

[0053] Furthermore, it is first necessary to clarify whether the reception mode is omnidirectional or directional. If it is directional, the angular range corresponding to the target pickup direction also needs to be determined. For omnidirectional reception, the spatial covariance matrix is ​​processed to ensure that the matrix can evenly reflect the audio signal characteristics in all directions, so that subsequent beamforming can cover all spatial directions. For directional reception, based on the target pickup direction, the signal weights in the corresponding direction in the spatial covariance matrix are strengthened, while the signal weights in non-target directions are weakened. Through such optimization and adjustment, an acoustic steering vector set of the original audio data that can accurately point to the target pickup direction is obtained, with each vector corresponding to an acoustic propagation characteristic in a specific direction.

[0054] Furthermore, by combining the acoustic steering vector set and the spatial covariance matrix, the audio signal intensity and noise interference in the direction corresponding to each acoustic steering vector are analyzed. For directional reception mode, the weight values ​​that enhance the signal in the target direction and suppress noise in non-target directions are calculated. For omnidirectional reception mode, the weight values ​​that amplify the effective signals in all directions and reduce environmental noise overall are calculated. These calculated weight values ​​are arranged in channel order to form the beamforming weight vector of the original audio data. This vector determines the subsequent adjustment amplitude of the audio signal in each channel.

[0055] Furthermore, based on the weight values ​​corresponding to each channel in the beamforming weight vector, the signal of each channel of the original audio data is adjusted. For channels with high weight values, the amplitude of their audio signals is amplified to highlight the effective sound; for channels with low weight values, the amplitude of their audio signals is reduced to suppress noise and interference. During the adjustment process, audio signals exceeding the spatial range corresponding to the configured recording mode are simultaneously filtered out to ensure that the processed audio signal retains only the effective sound within the target spatial range, ultimately yielding the mode-specific audio data of the original audio data.

[0056] Specifically, the spatial covariance matrix comes from the multi-channel spatial characteristic analysis of the original audio data. First, the propagation direction, phase difference and amplitude change information of each channel audio signal in the original audio data are extracted. The correlation between signals in different channels is analyzed, and the degree of mutual influence of each channel signal at different time points is calculated. Then, these analysis results are organized in matrix form, and each element in the matrix corresponds to the correlation between two channel signals. Finally, the spatial covariance matrix is ​​obtained.

[0057] Furthermore, the acoustic steering vector set is obtained by beamforming optimization of the spatial covariance matrix based on the configured sound pickup mode. If the configured sound pickup mode is omnidirectional, the spatial covariance matrix is ​​processed to evenly reflect the audio signal characteristics of each direction, forming an acoustic steering vector covering all spatial directions. If it is directional, the signal weights of the corresponding directions in the spatial covariance matrix are strengthened according to the target sound pickup direction, while the signal weights of non-target directions are weakened, thus obtaining an acoustic steering vector set that accurately points to the target sound pickup direction. Each vector corresponds to the acoustic propagation characteristics of a specific direction.

[0058] Furthermore, the conjugate transpose operator is a pre-defined standard operation rule in matrix operations, used to process the acoustic steering vector set. By keeping the real part of each element in the acoustic steering vector set unchanged and taking the opposite value of the imaginary part, and then performing row and column transformations on the processed vector set, the conjugate transpose of the acoustic steering vector set is obtained. This operation rule is a common and fixed operation method in the field of matrix operations, ensuring the standardization and accuracy of the operation process.

[0059] Furthermore, the regularization coefficient is a fixed value preset to avoid numerical instability during the inversion of the spatial covariance matrix. It is determined based on test data from a large number of audio processing scenarios and stored in the device's parameter configuration module. During the calculation process, it is used to adjust the matrix operation results to prevent large deviations in the calculation results due to small fluctuations in data, thus ensuring the stability of beamforming weight vector calculation.

[0060] Furthermore, the identity matrix is ​​a pre-defined standard matrix in matrix operations, in which the elements on the main diagonal are all specific fixed values, and the elements in the remaining positions are all zero. The structure and element values ​​of this matrix are universal and fixed in the field of matrix operations. It is used in conjunction with regularization coefficients during the calculation process to further optimize the matrix operation process and ensure the rationality and effectiveness of the calculation results.

[0061] Furthermore, the matrix inversion operator is a pre-defined standard operation rule in matrix operations, used to invert the spatial covariance matrix. Through specific matrix transformation steps, the spatial covariance matrix is ​​converted into its inverse matrix, so that multiplying the inverse matrix with the original spatial covariance matrix yields the identity matrix. This operation rule is a common and fixed operation method in the field of matrix operations, providing the necessary matrix form for the subsequent calculation of beamforming weight vectors.

[0062] Furthermore, the significance of this formula lies in its ability to accurately calculate the beamforming weight vector by integrating the characteristics of the spatial covariance matrix and the acoustic steering vector set, combined with regularization processing, thus providing a reliable basis for spatial filtering of the original audio data. The calculation involves first inverting the spatial covariance matrix to obtain its inverse; then multiplying this inverse matrix by the acoustic steering vector set to obtain the first intermediate calculation result; next, performing a conjugate transpose operation on the acoustic steering vector set, multiplying the result by the inverse matrix of the spatial covariance matrix, and then multiplying again by the acoustic steering vector set to obtain the second intermediate calculation result; then multiplying the regularization coefficient by the identity matrix to obtain the third intermediate calculation result; adding the second and third intermediate calculation results to obtain the denominator; and finally dividing the first intermediate calculation result by the denominator to obtain the beamforming weight vector.

[0063] Furthermore, this calculation method considers both the spatial distribution characteristics of the original audio data and the acoustic characteristics of the target reception direction. It also avoids numerical instability through regularization, ensuring that the beamforming weight vector can accurately adjust the audio signals of each channel. This enables subsequent spatial filtering to precisely retain effective sound from the target direction and suppress noise from non-target directions, ultimately obtaining mode-specific audio data that conforms to the configured reception mode.

[0064] Specifically, to eliminate phase deviations in the audio signals of different channels in an audio acquisition device, the original audio signals acquired from each channel are first obtained. By comparing the phase values ​​of the signals from different channels at the same time point, the phase differences between the channels are identified. Based on these phase differences, phase compensation adjustment is performed on the audio signals of each channel so that the audio signals of all channels reach a consistent phase state at the same time point. After this adjustment process, a coherent multi-channel signal of the original audio data is obtained.

[0065] Furthermore, when determining the spatial response characteristics of the microphone array based on the preset target pickup direction information, the angular range corresponding to the target pickup direction is first defined, such as the angles corresponding to the seats and positions specified in directional sound reception mode. Combined with the physical layout of the microphone array, the microphone array's sensitivity to audio signals, signal attenuation, and signal propagation delay within this target angular range are analyzed. These analytical results are then integrated to form the spatial response characteristics that reflect the signal reception characteristics of the microphone array in the target pickup direction.

[0066] Furthermore, when generating the corresponding beamforming weighting coefficients based on the spatial response characteristics, the signal strength characteristics of the target pickup direction and the noise distribution characteristics of the non-target directions are first analyzed. For the target pickup direction, a higher weighting coefficient is set to enhance signal reception in that direction; for the non-target directions, a lower weighting coefficient is set to reduce noise interference in that direction. Based on the specific signal strength and noise interference conditions, the weighting value corresponding to each microphone channel is determined, and these values ​​are arranged in channel order to form the beamforming weighting coefficients.

[0067] Furthermore, when performing spatial filtering on the coherent multi-channel signal based on beamforming weighting coefficients, the coherent multi-channel signal of each channel is multiplied by the corresponding beamforming weighting coefficient, amplifying the signal of the channel in the target pickup direction and reducing the signal of the channel in the non-target direction. Simultaneously, based on the spatial response characteristics of the microphone array, signals exceeding the angular range of the target pickup direction are filtered out, concentrating the processed signal in the target pickup direction, thereby obtaining a coherent multi-channel signal directional pickup beam.

[0068] Furthermore, when suppressing residual noise in the directional pickup beam, residual noise components other than the target speech signal, such as background noise in the environment, are first identified in the directional pickup beam. A noise suppression algorithm is then employed to attenuate the noise components in the directional pickup beam by analyzing the frequency characteristics and amplitude variation patterns of the residual noise, while preserving the integrity and clarity of the target speech signal. After noise suppression processing, pattern-specific audio data of the original audio data is generated.

[0069] Specifically, when acquiring the channel audio signals of the microphone array in the audio acquisition device, the device's audio acquisition function is activated. Each microphone in the microphone array acquires the sound signal at its location, which includes the target speech, ambient noise, and other content. The device categorizes and stores the sound signals acquired by each microphone according to their corresponding channels, ensuring that the audio signal of each channel is independent and complete, ultimately acquiring the audio signals of all channels of the microphone array in the audio acquisition device.

[0070] Furthermore, when performing inter-channel phase consistency analysis on the channel audio signals, signal segments at the same time point from all channel audio signals are selected as the analysis objects. The phase states of the signals from different channels at that time point are compared, and the difference between the phase of each channel and the phase of a preset reference channel is recorded. At the same time, the changing patterns of the phase differences between channels at different time points are observed. These phase differences and changing patterns are then organized to form the phase deviation characteristics of the channel audio signals that reflect the phase differences between channels.

[0071] Furthermore, when eliminating phase deviations between channel audio signals based on phase deviation characteristics, a corresponding phase compensation value is set for each channel's audio signal according to the phase difference between each channel and the reference channel in the phase deviation characteristics. The audio signal of each channel is adjusted segment by segment to ensure that the phase state of each channel is consistent with the reference channel at the same time point, ensuring synchronization of the audio signals of each channel in terms of propagation time and phase response. After channel-by-channel and segment-by-segment phase adjustment, a coherent multi-channel signal of channel audio signals with consistent phase is obtained.

[0072] In summary, multi-channel spatial characteristic analysis of the raw audio data generates a spatial covariance matrix, which comprehensively reflects the propagation direction, phase difference, and correlation of audio signals in each channel. This provides accurate spatial audio data support for subsequent beamforming processing and avoids processing deviations caused by incomplete spatial feature extraction.

[0073] In summary, the acoustic steering vector set is obtained by optimizing the spatial covariance matrix based on the configured radio mode. In directional mode, the signal weight of the target pickup direction can be enhanced, and in omnidirectional mode, the characteristics of each direction can be balanced, so that the beamforming can accurately match the radio reception needs of the business scenario and solve the limitations of the single radio reception mode of traditional equipment.

[0074] In summary, when calculating the beamforming weight vector formula, a regularization coefficient and an identity matrix are introduced to avoid numerical instability issues when inverting the spatial covariance matrix. This ensures that the weight vector can accurately adjust the signal amplitude of each channel, enhance the effective sound in the target direction and suppress noise in the non-target direction in directional mode, and amplify the effective signals in each direction in an equal manner in omnidirectional mode, thereby improving the stability of audio processing.

[0075] In summary, spatial filtering based on weight vectors filters out interference signals that exceed the range of the configured audio reception mode. In directional mode, it generates directional audio with precise pointing and low noise, while in omnidirectional mode, it generates omnidirectional audio with comprehensive coverage. This provides high-quality data for subsequent multi-level audio enhancement, meeting the needs of medical aesthetics, business halls, and other scenarios for high-quality audio sources, and improving the accuracy of subsequent speech recognition and transcription.

[0076] In summary, by eliminating the phase deviation of the audio signals in each channel of the audio acquisition device, the multi-channel signals are synchronized in phase, resulting in coherent multi-channel signals. This avoids audio signal superposition distortion caused by phase inconsistency between channels, laying the foundation for subsequent precise spatial filtering and meeting the signal consistency requirements of microphone arrays.

[0077] In summary, by determining the spatial response characteristics of the microphone array based on the preset target pickup direction, clarifying the signal reception sensitivity and attenuation law of the array in the target direction, and ensuring that the beamforming weighting coefficients generated subsequently can accurately point to the target direction, the problem of fixed pickup direction of traditional equipment and inability to adapt to the target needs of multiple scenarios is solved.

[0078] In summary, spatial filtering of coherent multi-channel signals based on beamforming weighting coefficients amplifies the effective sound in the target pickup direction and weakens interference signals in non-target directions, forming a directional pickup beam. For example, bidirectional directional pickup can simultaneously collect the sound of the agent's seat and the counter, meeting the bidirectional sound reception needs of scenarios such as counters and increasing the proportion of the target audio signal.

[0079] In summary, the residual noise in the directional pickup beam is suppressed, the audio signal is further purified, and mode-specific audio data is generated to ensure the audio signal-to-noise ratio and clarity. This provides a high-quality audio source for subsequent multi-level audio enhancement, role separation and ASR transcription, improves the accuracy of audio analysis in business systems, and meets the needs of medical aesthetics, business halls and other scenarios for high-quality recording.

[0080] In summary, when acquiring the channel audio signals of the microphone array, it is essential to ensure that the sound signals collected by each microphone are stored independently and completely to avoid affecting subsequent processing due to signal loss or omission. This provides comprehensive raw data for phase deviation analysis and meets the hardware configuration requirements of the device's multi-channel sound pickup.

[0081] In summary, performing phase consistency analysis on the channel audio signal involves comparing the phase states of different channels at the same time point, recording the phase difference and its variation patterns, generating phase deviation characteristics, clearly locating phase inconsistency issues between channels, avoiding signal distortion caused by blind adjustments, and providing a precise basis for subsequent deviation elimination.

[0082] In summary, based on phase deviation characteristics, phase compensation values ​​are set for each channel, and the phase of the audio signal in each channel is adjusted segment by segment to ensure that the phase of all channel signals is consistent at the same time point, resulting in a coherent multi-channel signal. This solves the phase deviation problem caused by microphone array hardware layout and signal propagation delay, avoids interference when multi-channel signals are superimposed, provides a high-quality synchronization signal for subsequent directional pickup beamforming and spatial filtering, and ensures the directivity and clarity of mode-specific audio data, meeting the needs of precise sound pickup in scenarios such as business halls and medical aesthetics clinics.

[0083] S4. Perform multi-level audio enhancement processing on the mode-specific audio data to obtain enhanced audio data of the mode-specific audio data; In this embodiment of the invention, the step of performing multi-level audio enhancement processing on the pattern-specific audio data to obtain enhanced audio data of the pattern-specific audio data includes: Eliminate the echo component of the mode-specific audio data to generate primary processed audio data of the mode-specific audio data; The primary processed audio data is subjected to noise suppression processing to obtain secondary processed audio data of the mode-specific audio data; The signal-to-noise ratio and clarity of the speech segments in the secondary processed audio data are improved to obtain enhanced audio data of the mode-specific audio data.

[0084] Specifically, when eliminating echo components in pattern-specific audio data, the echo signals generated by environmental reflections in the audio are first identified. By comparing the differences in time delay and amplitude attenuation between the original speech signal and the reflected echo signal in the pattern-specific audio data, the characteristic parameters of the echo are determined. Based on these characteristic parameters, a cancellation signal with the same amplitude but opposite phase to the echo signal is generated. This cancellation signal is then superimposed on the echo portion in the pattern-specific audio data, causing them to cancel each other out, thereby removing the echo components from the audio and generating the primary processed audio data for the pattern-specific audio data.

[0085] Furthermore, when performing noise suppression on the primary processed audio data, the frequency distribution and amplitude variation patterns of environmental noise in the primary processed audio data are first analyzed to distinguish different frequency bands of the speech signal and the noise signal. For frequency bands where noise is concentrated, filtering is used to weaken the signal amplitude within that band while maintaining the integrity of the main frequency bands of the speech signal. During the processing, changes in the audio signal are monitored in real time to ensure that noise suppression does not cause excessive attenuation of the speech signal. After this processing, secondary processed audio data of pattern-specific audio data is obtained.

[0086] Furthermore, when improving the signal-to-noise ratio and clarity of speech segments in the secondary-processed audio data, the speech segments are first located, and key features such as the fundamental frequency and harmonics of the speech signal are identified. For each speech segment, the intensity of the speech signal is increased by enhancing the amplitude of the fundamental frequency signal and major harmonic components, while further filtering out residual weak noise in the speech segment. In addition, the transition parts of the speech signal are smoothed to reduce distortion caused by signal abrupt changes, making the speech sound more coherent and clear. After these processes, the effective signal of the speech segment is enhanced, noise is further reduced, and finally, enhanced audio data specific to the pattern is obtained.

[0087] In summary, by eliminating echo components in pattern-specific audio data, primary processed audio data is generated, avoiding speech overlap and blurring caused by echoes. This lays a clean audio foundation for subsequent noise suppression and clarity improvement, meeting the basic requirements of business scenarios for audio purity.

[0088] In summary, reducing environmental noise and improving speech recognition accuracy involves: suppressing noise in the primary processed audio data, specifically reducing background noise in the environment, preserving the integrity of the main frequency bands of the target speech signal, generating secondary processed audio data, reducing noise interference with the speech signal, and enabling subsequent speech recognition and transcription to capture effective information more accurately.

[0089] In summary, improving the signal-to-noise ratio and intelligibility of speech segments in secondary processed audio data is achieved by enhancing the fundamental frequency and harmonic components of the speech, smoothing signal transitions, strengthening speaker voice characteristics, and reducing signal distortion, ultimately resulting in enhanced audio data. This data can meet the high-quality audio source requirements of backend role separation, voiceprint recognition, and ASR transcription services, improve the accuracy of audio analysis in business systems, and align with the core objective of audio pickup devices to provide high-quality audio sources for business systems.

[0090] In summary, the multi-level enhancement processing is optimized step by step in the order of "re-echo removal - noise suppression - clarity enhancement". It not only comprehensively solves different types of interference problems in audio, but also adapts to the audio characteristics of different business scenarios such as two-way sound pickup in business halls and close-range communication in medical aesthetics scenarios. It ensures that the enhanced audio data output in various scenarios has high availability and avoids the limitations of a single processing method.

[0091] S5. The enhanced audio data is transmitted to a remote server, and the enhanced audio data is transcribed into a role-based structure to obtain the voiceprint identifier text data of the enhanced audio data.

[0092] In this embodiment of the invention, the step of transmitting the enhanced audio data to a remote server and performing role-based structured transcription on the enhanced audio data to obtain the voiceprint identifier text data of the enhanced audio data includes: The enhanced audio data is transmitted to the data receiving end of the remote server, and a secure transmission confirmation signal for the enhanced audio data is generated. In response to the secure transmission confirmation signal, voice activity detection is performed on the enhanced audio data to obtain the time stamp information of the enhanced audio data; Based on the time stamp information, extract the voiceprint feature vector of the detected speech segment; The voiceprint feature vectors are clustered and grouped according to the speech segments of different speakers to obtain the role-separated audio data of the voiceprint feature vectors; The character-separated audio data is converted into text data to obtain the original transcribed text of the character-separated audio data; Based on the voiceprint feature vector and the time stamp information, the original transcribed text is associated and fused with the corresponding speaker identity to obtain the voiceprint identifier text data of the enhanced audio data.

[0093] Specifically, when transmitting enhanced audio data to the data receiving end of the remote server, an encrypted transmission channel is initiated between the device and the remote server. This channel employs a preset secure transmission protocol to ensure that data transmission is not stolen or tampered with during the transmission process. The device sends the enhanced audio data to the data receiving end of the remote server in fixed data block sizes. Each time the remote server receives a data block, it sends a reception confirmation message back to the device until all data blocks have been transmitted. Once the device confirms that all enhanced audio data has been successfully received by the remote server, it generates a secure transmission confirmation signal for the enhanced audio data. This signal includes information such as the data transmission completion status and data integrity verification results.

[0094] Furthermore, upon receiving the secure transmission confirmation signal, the remote server immediately activates the voice activity detection function, analyzing the enhanced audio data segment by segment to identify segments containing voice signals and silent segments without voice signals. For the identified voice segments, the start and end times in the enhanced audio data are recorded; for silent segments, their corresponding time intervals are also marked. The time information of these voice and silent segments is then organized according to the audio playback order to form time-stamped information for the enhanced audio data that clearly indicates the distribution of voice signals on the time axis.

[0095] Furthermore, when extracting the voiceprint feature vector of the detected speech segments based on the time stamp information, all detected speech segments are first accurately extracted from the enhanced audio data according to the start and end times of the speech segments in the time stamp information. Feature extraction is performed on each detected speech segment, analyzing the unique acoustic features of the speaker in the speech segment, such as pitch, timbre, and speech rate. These acoustic features are converted into vector forms that quantify the voiceprint characteristics. Each detected speech segment corresponds to a unique voiceprint feature vector, and finally, the voiceprint feature vectors of all detected speech segments are obtained.

[0096] Furthermore, when clustering the voiceprint feature vectors according to the speech segments of different speakers, the similarity between all voiceprint feature vectors is compared, and the degree of difference between different vectors in quantitative features such as pitch and timbre is calculated. Voiceprint feature vectors with a difference less than a preset standard are grouped into the same group, and it is believed that the speech segments corresponding to these vectors come from the same speaker; voiceprint feature vectors with a difference greater than the preset standard are divided into different groups, and it is determined that they correspond to different speakers. After multiple comparisons and classification adjustments, it is ensured that the voiceprint feature vectors within the same group are highly similar and the vectors between different groups are significantly different. Finally, role-separated audio data of voiceprint feature vectors are obtained, with each group of data corresponding to the speech information of one speaker.

[0097] Furthermore, when converting the role-separated audio data into text data, speech recognition processing is performed on the speech segments corresponding to each speaker in the role-separated audio data to identify the word content and grammatical structure of the speech segments. Based on the speech recognition results, the continuous speech signals are converted into corresponding text forms sentence by sentence, while correcting any word errors or grammatical inconsistencies that may occur during the recognition process, ensuring that the converted text accurately reflects the original meaning of the speech segments. The converted texts of all speech segments corresponding to each speaker are integrated to form the original transcribed text of the role-separated audio data, with each original transcribed text corresponding one-to-one with the corresponding speaker group.

[0098] Furthermore, when associating and fusing the original transcribed text with the corresponding speaker identity based on voiceprint feature vectors and time stamp information, the unique identifier of the voiceprint feature vector corresponding to each speaker group is first determined, and then the time interval corresponding to each speech segment in the time stamp information is extracted. Each sentence in the original transcribed text is bound to the corresponding speech segment time interval and the voiceprint feature vector identifier of the speaker group to which it belongs, so that each sentence can be clearly associated with a specific speaker and the specific time position of the sentence in the audio. All binding information is integrated to form structured data containing speaker identity identifiers, sentence text, and time intervals, ultimately obtaining voiceprint-identified text data for enhanced audio data.

[0099] In summary, the enhanced audio data is transmitted to a remote server through an encrypted transmission channel. Each data block transmitted is acknowledged upon receipt, and a secure transmission confirmation signal is generated in the end. This ensures that the audio data is not stolen, tampered with, or lost during transmission, providing a complete data source for subsequent transcription processing. This meets the security requirements of recording data in scenarios such as business halls and medical aesthetics clinics.

[0100] In summary, after responding to the secure transmission confirmation signal, the system performs voice activity detection on the enhanced audio data, marks the time intervals of voice segments and silence segments to generate time stamp information, accurately extracts valid voice segments, avoids invalid processing of silent parts, improves the efficiency of subsequent voiceprint extraction and transcription, and meets the business system's requirements for the timeliness of audio processing.

[0101] In summary, by extracting voiceprint feature vectors from speech segments based on time stamp information and grouping them by similarity to obtain role-separated audio data, we can effectively distinguish the speech of different speakers, solve the problem that traditional transcription cannot separate multiple roles, and adapt to the audio parsing needs of multi-role interaction scenarios.

[0102] In summary, after transcribing the voice data of the separated characters into raw text, the voiceprint feature vector and time stamp information are associated, binding the text with the corresponding speaker's identity and audio time position to form voiceprint-identified text data. This data has a high degree of structure, which facilitates the backend quality inspection system to quickly locate the text content corresponding to the voice of a specific character, improving the efficiency of recording quality inspection and business analysis, and meeting the core needs of business systems for accurate analysis and efficient utilization of audio data.

[0103] like Figure 2 The diagram shown is a functional block diagram of a face detection real-time synchronous audio data processing system provided in an embodiment of the present invention.

[0104] The face detection real-time synchronous audio data processing system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the face detection real-time synchronous audio data processing system 100 may include an effective face detection module 101, an intelligent recording trigger module 102, an adaptive beamforming module 103, a multi-level audio enhancement module 104, and a character structure transcription module 105. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0105] In this embodiment, the functions of each module / unit are as follows: The effective face detection module 101 is used to perform face detection processing on the acquired video stream data. When a face that matches the preset pose features is detected within a preset distance range, a timer is started, and when the continuous detection time reaches a preset threshold, an effective face detection signal of the video stream data is generated. The intelligent recording trigger module 102 is used to respond to the valid face detection signal. When the signal duration reaches the preset threshold, it automatically triggers the sound pickup device to start recording and synchronously controls the indicator light status to generate the original audio data of the video stream data. The adaptive beamforming module 103 is used to select the corresponding beamforming scheme according to the preset configuration reception mode, and perform adaptive beamforming processing on the original audio data to obtain mode-specific audio data of the original audio data. The multi-level audio enhancement module 104 is used to perform multi-level audio enhancement processing on the mode-specific audio data to obtain enhanced audio data of the mode-specific audio data. The character structure transcription module 105 is used to transmit the enhanced audio data to a remote server and perform character structure transcription on the enhanced audio data to obtain the voiceprint identifier text data of the enhanced audio data.

[0106] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0107] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0109] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0110] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A face detection real-time synchronous sound pickup data processing method, characterized in that, The method comprises: S1, face detection processing is performed on the collected video stream data, timing is started when a face conforming to a preset posture feature is detected within a preset distance range, and an effective face detection signal of the video stream data is generated when a continuous detection time reaches a preset threshold; S2, in response to the effective face detection signal, a pickup device is automatically triggered to start recording and a status of an indicator light is synchronously controlled when a signal duration reaches the preset threshold, raw audio data of the video stream data is generated, including: confidence factor of the effective face detection signal is obtained by confidence evaluation of the effective face detection signal, wherein a calculation formula of the confidence factor is: ； In the formula, is a comprehensive confidence factor, is a time weight coefficient, is a signal duration, is a preset time threshold, is an intensity weight coefficient, is a current signal intensity, is a maximum signal intensity, is a historical confidence mean value, is a previous confidence value, is a fluctuation weight coefficient, is a signal change variance, is an absolute value of a characteristic difference between the historical confidence mean value and the previous confidence value, is a continuity weight coefficient; According to the quantitative comparison of the confidence factor and the preset threshold, the confidence factor is mapped to the corresponding confidence level to generate a hierarchical triggering instruction of the video stream data; Based on the hierarchical triggering instruction, multi-level response is performed on the video stream data to obtain hierarchical response video data of the video stream data; According to the hierarchical response video data, raw audio data of the video stream data is generated; S3, according to a preset configuration of a sound collecting mode, a corresponding beam forming scheme is selected, and adaptive beam forming processing is performed on the raw audio data to obtain mode-specific audio data of the raw audio data; S4, multi-level audio enhancement processing is performed on the mode-specific audio data to obtain enhanced audio data of the mode-specific audio data; S5, the enhanced audio data is transmitted to a remote server, and role structured transcription is performed on the enhanced audio data to obtain voiceprint identification text data of the enhanced audio data.

2. The real-time face detection and data processing method of claim 1, wherein, The face detection processing on the collected video stream data, when a face conforming to a preset posture feature is detected within a preset distance range, timing is started, and an effective face detection signal of the video stream data is generated when a continuous detection time reaches a preset threshold, comprises: Multi-posture face feature analysis is performed on the video stream data to obtain key feature information of the video stream data; The matching degree of the key feature information and the preset posture feature is analyzed, and timing is started when the matching degree continuously reaches a preset threshold; The key feature information is verified for consistency between continuous frames to ensure stability of the posture feature; When the verification for consistency between continuous frames is passed and a duration reaches the preset threshold, an effective face detection signal of the video stream data is generated.

3. The real-time face detection and data processing method of claim 1, wherein, The adaptive beam forming processing is performed on the raw audio data according to the preset configuration of the sound collecting mode to obtain mode-specific audio data of the raw audio data, comprising: Multi-channel spatial characteristic analysis is performed on the raw audio data to obtain a spatial covariance matrix of the raw audio data; Based on the configuration of the sound collecting mode, beam forming optimization is performed on the spatial covariance matrix to obtain an acoustic steering vector set of the raw audio data; A beam forming weight vector of the raw audio data is calculated, wherein a calculation formula of the beam forming weight vector is: ； wherein is the beamforming weight vector, is the spatial covariance matrix, is a set of acoustic steering vectors, is a conjugate transpose operator, is a regularization coefficient, is an identity matrix, is a matrix inversion operator; Based on the beam forming weight vector, spatial domain filtering processing is performed on the raw audio data to obtain mode-specific audio data of the raw audio data.

4. The real-time face detection and data processing method of claim 3, wherein, The spatial domain filtering processing of the original audio data based on the beamforming weight vector comprises: Eliminating the phase deviation of the channel audio signals in the audio acquisition device to obtain the coherent multi-channel signals of the original audio data; According to the preset target pickup direction information, the spatial response characteristics of the microphone array are determined; According to the spatial response characteristics, the corresponding beamforming weight coefficients are generated; Based on the beamforming weight coefficients, the spatial filtering processing of the coherent multi-channel signals is performed to obtain the directional pickup beam of the coherent multi-channel signals; Residual noise of the directional pickup beam is suppressed to generate the mode-specific audio data of the original audio data.

5. The real-time face detection and data processing method of claim 4, wherein, The elimination of the phase deviation of the channel audio signals in the audio acquisition device to obtain the coherent multi-channel signals of the original audio data comprises: Obtaining the channel audio signals of the microphone array in the audio acquisition device; Inter-channel phase consistency analysis is performed on the channel audio signals to obtain the phase deviation characteristics of the channel audio signals; Based on the phase deviation characteristics, the phase deviation between the channel audio signals is eliminated to obtain the coherent multi-channel signals of the channel audio signals.

6. The real-time face detection and data processing method of claim 1, wherein, The multi-level audio enhancement processing of the mode-specific audio data to obtain the enhanced audio data of the mode-specific audio data comprises: Eliminating the echo component of the mode-specific audio data to generate the primary processing audio data of the mode-specific audio data; The noise suppression processing of the primary processing audio data is performed to obtain the secondary processing audio data of the mode-specific audio data; The signal-to-noise ratio and intelligibility of the speech segment in the secondary processing audio data are improved to obtain the enhanced audio data of the mode-specific audio data.

7. The real-time face detection and data processing method of claim 1, wherein, The enhanced audio data is transmitted to a remote server, and the role structured transcription of the enhanced audio data is performed to obtain the voiceprint identification text data of the enhanced audio data, comprising: The enhanced audio data is transmitted to the data receiving end of the remote server to generate a secure transmission confirmation signal of the enhanced audio data; In response to the secure transmission confirmation signal, the voice activity detection of the enhanced audio data is performed to obtain the time mark information of the enhanced audio data; According to the time mark information, the voiceprint feature vector of the detected speech segment is extracted; According to the voice segment of different speakers, the voiceprint feature vectors are clustered and grouped to obtain the role-separated audio data of the voiceprint feature vectors; The role-separated audio data is converted into text data to obtain the original transcription text of the role-separated audio data; Based on the voiceprint feature vectors and the time mark information, the original transcription text and the corresponding speaker identity are associated and fused to obtain the voiceprint identification text data of the enhanced audio data.

8. A face detection real-time synchronous sound pickup data processing system, characterized in that, The system for using the real-time synchronous pickup data processing method for face detection according to claim 1, the system comprises: An effective face detection module is configured to perform face detection on the collected video stream data, start timing when a face conforming to a preset posture feature is detected within a preset distance range, and generate an effective face detection signal of the video stream data when the continuous detection time reaches a preset threshold; An intelligent recording trigger module is configured to respond to the effective face detection signal, automatically trigger a sound pickup device to start recording and synchronously control an indicator light state when the signal duration reaches the preset threshold, and generate original audio data of the video stream data; An adaptive beamforming module is configured to select a corresponding beamforming scheme according to a preset configuration sound collection mode, perform adaptive beamforming processing on the original audio data, and obtain mode-specific audio data of the original audio data; A multi-level audio enhancement module is configured to perform multi-level audio enhancement processing on the mode-specific audio data, and obtain enhanced audio data of the mode-specific audio data; A role structured transcription module is configured to transmit the enhanced audio data to a remote server, perform role structured transcription on the enhanced audio data, and obtain voiceprint identification text data of the enhanced audio data.

Images