A camera-based daily language quantity automatic counting and voiceprint privacy protection method

Abstract

The present application relates to the field of artificial intelligence and intelligent sensing technology, in particular to a daily language quantity automatic counting and voiceprint privacy protection method based on a camera, comprising the following steps: S1, collecting facial features, mouth shapes and voice features, fusing visual and voice information to obtain speaker identity and voice period; S2, generating anonymized voice through X-vector replacement and noise disturbance on the edge device; S3, transcribing, word segmentation and switching detection on the anonymized voice, generating language output quantity and encrypting storage. The present application, through multi-modal collaborative recognition of camera and microphone array, edge side voiceprint anonymization, multi-dimensional statistics and encrypted storage of daily language output quantity, improves the accuracy of speaker recognition and language output quantity statistics, effectively reduces the risk of voiceprint leakage and guarantees the security and integrity of statistical data.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent sensing technology, and in particular to a method for automatically counting daily speech volume and protecting voiceprint privacy based on a camera. Background Technology

[0002] With the development of smart homes, children's language development assessment, elderly cognitive health monitoring, and human-computer interaction technologies, long-term, continuous, and objective quantitative analysis of users' daily language behavior has become an important research direction in the field of smart health monitoring. Existing language monitoring systems typically use sensing devices such as cameras and microphones to collect users' visual and speech information, and obtain information about users' language activities through technologies such as speech recognition, speaker recognition, and language statistics, providing data support for language ability assessment, rehabilitation training, and health management.

[0003] However, most existing technologies rely on a single voice signal to complete speaker recognition and speech volume statistics. In scenarios such as multi-person dialogues, complex backgrounds, low light, and target occlusion, problems such as speaker recognition errors, inaccurate speech attribution, and speech volume statistics bias are prone to occur. At the same time, existing solutions usually directly process or store voice data containing voiceprint information. Voiceprint features are strongly correlated with user identity, posing a significant risk of privacy leakage during voice transmission, storage, and analysis. Moreover, most solutions lack data processing mechanisms that balance the accuracy of speech statistics with the protection of voiceprint privacy, making it difficult to meet users' dual needs for language behavior analysis and personal privacy security. Summary of the Invention

[0004] This invention provides a method for automatic counting of daily speech output and voiceprint privacy protection based on a camera. By fusing facial features and lip movement features captured by the camera with multi-channel speech signals captured by a microphone array, speaker detection and identity recognition are achieved. Furthermore, X-vector replacement and dynamic noise perturbation techniques are used to complete voiceprint anonymization processing. Combined with speech recognition, text word count, speaker switch detection, and phoneme-level analysis, the method calculates the user's daily speech output. At the same time, PBKDF2, HKDF, and AES-GCM are used to encrypt and store the statistical results, thereby improving the accuracy of daily speech output statistics while effectively protecting the user's voiceprint privacy and the security of statistical data.

[0005] A method for automatically counting everyday speech volume and protecting voiceprint privacy based on a camera includes the following steps: S1: The system collects user facial features and lip movement features in a non-contact manner through a camera, and collects multi-channel speech signals through a microphone array. It uses lip movement detection and facial feature recognition technology of the camera to detect speaking behavior. When a speaker is detected, the microphone array is triggered to collect speech data. The facial features, lip movement features and multi-channel speech signals are jointly processed. The lip trajectory and speech frame are aligned in time through a spatiotemporal convolutional network and a lightweight Transformer. The speaker's identity identifier and speech start and end timestamps are output to obtain the speaker detection and identity recognition results. S2, based on the speaker detection and identity recognition results, the multi-channel speech signal is subjected to voiceprint feature stripping and anonymization processing at the edge device end. X-vector replacement and dynamic noise perturbation technology are used to perturb the voiceprint features in the speech data to generate anonymized speech data. S3, based on anonymized speech data, uses a speech recognition model to transcribe the anonymized speech. Based on the transcribed text, it counts the total number of words in the text content through Chinese word segmentation, and combines the speaker switching detection results to count the number of times the speaker speaks. At the same time, it estimates the user's daily effective language output based on the phoneme-level duration, pause frequency, and speech rate changes in the anonymized speech data, and generates a daily language output statistics result. The daily language output statistics result is encrypted using the AES-GCM algorithm, and the corresponding encryption key is generated through the PBKDF2 and HKDF key derivation algorithms.

[0006] Optionally, S1 includes: S11, continuously capture user facial videos in a non-contact manner using a camera, perform face region detection on the captured video frames, enhance the video frames when the ambient light intensity is lower than a preset light threshold, extract user facial features within the face region, and extract lip key points in each video frame using a lip movement detection model. Calculate the lip movement amplitude based on the positional changes of lip key points between adjacent video frames. When the lip movement amplitude reaches a preset lip movement judgment threshold, mark the corresponding user as a candidate speaker and obtain the candidate speaker's facial features and lip movement features. S12, when a candidate speaker is detected, the microphone array is triggered to collect multi-channel speech signals. The speech signals of each microphone channel are localized and beamformed to generate a speech signal corresponding to the candidate speaker. Voiceprint features are extracted from the speech signals, and facial features and lip movement features are extracted from the camera acquisition results. The fusion weight of visual features and voiceprint features is dynamically adjusted according to the video occlusion rate and speech noise intensity. The multimodal identity matching value of the candidate speaker is calculated. The candidate speaker whose multimodal identity matching value reaches the preset identity matching threshold is determined as the target speaker. When the user is facing away from or to the side of the camera and cannot obtain effective visual features, the visual feature weight is adjusted to zero, and the target speaker is determined based on the voiceprint features. S13, the lip movement features of the target speaker are encoded into a visual temporal feature sequence, and the multi-channel speech signal corresponding to the target speaker is divided into continuous speech frames. The lip movement trajectory features and speech frame features are extracted by a spatiotemporal convolutional network. The lip movement trajectory features and speech frame features are then input into a lightweight Transformer for cross-modal association. The temporal correspondence between each lip movement segment and the corresponding speech frame is determined based on the cross-modal association features. The start time and end time of the speech are identified, and the identity information of the target speaker is bound to the start time and end time of the speech. The speaker identity identifier and the start and end timestamps of the speech are output, and the speaker detection and identity recognition results are generated.

[0007] Optionally, S11 includes: S111, continuously capture user facial video using a camera at a preset sampling frequency, decompose the user facial video into video frames arranged in chronological order, and calculate the first... The ambient light intensity of each video frame is used. When the ambient light intensity is lower than the preset light threshold, the Retinex algorithm is used to enhance the video frame, and the enhanced video frame or the video frame whose light intensity reaches the preset light threshold is used as the video frame to be detected. S112, the video frame to be detected is input into the face feature extraction model to detect the face position in the video frame, determine the corresponding face region, and extract facial features used to represent user identity within the face region. The face region is then input into the lip movement detection model to locate the lip contour and extract the first... The set of lip key points in each video frame is arranged continuously according to the time sequence of the video frames to form the user's lip key point time sequence data. S113, based on the temporal data of lip key points, calculate the positional changes of corresponding lip key points between adjacent video frames, and obtain the first... The lip movement amplitude corresponding to each video frame is used to mark the user in the corresponding video frame as a candidate speaker when the lip movement amplitude reaches the preset lip movement judgment threshold. The facial features extracted from the face area corresponding to the candidate speaker and the changes in the position of the key points of the lips in the continuous video frames are used as the facial features and lip movement features of the candidate speaker.

[0008] Optionally, S12 includes: S121, when a candidate speaker is detected, trigger the... A microphone array consisting of several microphones synchronously acquires multi-channel speech signals, denoted as follows: ,in, Calculate the generalized cross-correlation function between any two microphone channels, determine the inter-channel time delay based on the maximum value of the generalized cross-correlation function, and determine the sound source direction based on the estimated inter-channel time delay and the positional relationship of each microphone in the microphone array. The system performs delay compensation and weighted superposition on the multi-channel speech signals according to the direction of the sound source, and maps the direction of the sound source to the position of the candidate speaker in the camera to generate a speech signal corresponding to the candidate speaker. S122, extract voiceprint features from the speech signal corresponding to the candidate speaker, and obtain facial features and lip movement features from the camera acquisition results. Use the facial features and lip movement features as the visual features of the candidate speaker, and calculate the visual feature matching value of the candidate speaker respectively. Matching values ​​of voiceprint features The reliability of visual features and the reliability of voiceprint features are calculated based on the video occlusion rate and speech noise intensity of the candidate speaker. The weights of visual features and voiceprint features are calculated based on the reliability of visual features and the reliability of voiceprint features. S123, based on visual feature matching values, voiceprint feature matching values, visual feature weights, and voiceprint feature weights, calculate the... The multimodal identity matching value of the candidate speaker is used to determine the next step. When the multimodal identity matching value reaches a preset identity matching threshold, the candidate speaker will be... When a candidate speaker is identified as the target speaker, and the candidate speaker is facing away from or to the side of the camera, making it impossible to effectively extract their facial features or lip movement features, the visual feature weight is set to 0 and the voiceprint feature weight is set to 1.

[0009] Optionally, S13 includes: S131, Arrange the key lip points and their positional changes of the target speaker in consecutive video frames according to the time sequence of the video frames, and then... The lip movement features corresponding to each video frame are encoded into a visual feature vector. This forms a visual temporal feature sequence. Simultaneously, the speech signal corresponding to the target speaker is divided into continuous speech frames according to a preset frame length and frame shift. Each speech frame is represented as This forms a sequence of speech frames; S132, the visual temporal feature sequence and the speech frame sequence are respectively input into the spatiotemporal convolutional network to extract the spatial morphological changes and temporal evolution of continuous mouth movements to obtain the lip movement trajectory feature sequence, and the temporal frequency changes of continuous speech frames are extracted to obtain the speech frame feature sequence. The lip movement trajectory feature sequence is used as query information, and the speech frame feature sequence is used as key and value input into the lightweight Transformer to calculate the cross-modal attention matrix. S133, based on the cross-modal correlation matrix, for the first For each lip movement segment, the speech frame with the highest correlation value is selected as the corresponding speech frame. The time corresponding to the first speech frame that continuously meets the correlation condition is determined as the speech start time, and the time corresponding to the last speech frame that continuously meets the correlation condition is determined as the speech end time. The identity information of the target speaker is bound to the speech start time and the speech end time respectively, and the speaker identity identifier and speech start and end timestamps are output to generate speaker detection and identity recognition results.

[0010] Optionally, S2 includes: S21, based on the speaker detection and identity recognition results, extract the multi-channel speech signal corresponding to the target speaker at the edge device. After preprocessing the multi-channel speech signal, input it into the ECAPA-TDNN model to extract the X-vector voiceprint feature vector of the target speaker, and calculate the cosine distance between the X-vector voiceprint feature vector and each pseudo-X-vector feature vector in the pre-trained X-vector library. ,when When the time comes, the corresponding pseudo-X-vector will be determined as the target replacement vector; S22, based on the first The X-vector voiceprint feature vector of a target speaker is selected to satisfy... The pseudo-X-vector feature vector is used as the target replacement vector to replace the X-vector voiceprint feature vector, generating an anonymized voiceprint feature vector. And based on the anonymized voiceprint feature vector, the multi-channel speech signal corresponding to the target speaker is processed by voiceprint feature replacement to generate initial anonymized speech data; S23. For the generated initial anonymized speech data, analyze its speech spectrum features, determine the perturbation weights corresponding to different spectrum regions based on the sensitivity of voiceprint features, perform dynamic noise perturbation on the speech spectrum features, and convert the perturbed speech spectrum features into a time-domain speech signal to obtain anonymized speech data.

[0011] Optionally, S3 includes: S31, based on the generated anonymized speech data, the speech content of the anonymized speech data is transcribed using a speech recognition model to generate speech-transcribed text, and a text word set is obtained through Chinese word segmentation. The number of words in the text word set is counted to generate text word count features. At the same time, the anonymized speech data is divided into speech segments. S32, based on speech-to-text, text word count features, and anonymized speech data, speaker switching detection technology is used to identify speaker switching positions in continuous speech segments and count the number of times the target speaker speaks. At the same time, phoneme-level analysis is performed on the anonymized speech data to extract phoneme-level duration, pause frequency, and speech rate change features. Combined with text word count, number of times spoken, effective language duration, pause frequency, and speech rate change features, the daily language output of the target speaker is calculated, and a daily language output statistical result is generated. S33. Based on the generated daily language output statistics, an initial key is generated by combining user password information and random salt value with the PBKDF2 key derivation algorithm, and the initial key is expanded using the HKDF key derivation algorithm to generate an AES-GCM encryption key. The daily language output statistics are then encrypted using the AES-GCM encryption key to generate encrypted daily language output statistics including encrypted data and integrity authentication information, thereby achieving secure storage of the daily language output statistics.

[0012] Optionally, S31 includes: S311, the first Anonymous speech data of a target speaker is input into a speech recognition model to identify the speech content in the speech signal and obtain the corresponding speech-to-text. The speech recognition model adopts the lightweight speech recognition model Distil-Whisper. S312, Based on the speech-to-text transcription, the speech-to-text is segmented into words using a Chinese word segmentation tool, the continuous text is converted into multiple independent word units, a text word set is generated, and the text word count feature is statistically analyzed based on the text word set; S313, based on the corresponding anonymized speech data, the continuous speech signal is segmented according to the speech activity detection results, the region with continuous speech activity is divided into independent speech segments, a speech segment set is generated, and the time interval is calculated according to the temporal continuity relationship between adjacent speech segments. When the time interval is not greater than the preset speech interval threshold, it is retained as an independent speech segment; otherwise, the speech segments are merged to finally generate a speech segment set.

[0013] Optionally, S32 includes: S321, Based on the generated set of speech segments, extract the semantic and acoustic features of adjacent speech segments respectively, and calculate the speaker switching determination value according to the semantic and acoustic differences between adjacent speech segments. ,when At that time, the first The first audio segment and the first The position between each audio segment is used to determine the speaker switching position. A preset speaker switching threshold is set, and the number of times the target speaker speaks is counted based on the identified speaker switching position; S322, perform phoneme-level alignment on the anonymized speech data, identify the start and end times of each effective phoneme, calculate the duration of each effective phoneme, accumulate the durations of all effective phonemes to obtain the effective speech duration, calculate the pause frequency based on the number of pause segments in the anonymized speech data whose duration reaches the preset pause duration threshold, calculate the speech rate based on the number of words in each continuous speech segment and the effective speech duration, and calculate the speech rate change characteristics between adjacent speech segments; S323, input text word count features, number of utterances, effective language duration, pause frequency, and speech rate variation features into the language quantity statistical model, and calculate the first... The daily language output of a target speaker is calculated, and the target speaker's identity, daily language output, and statistical time are linked to generate statistical results of daily language output.

[0014] Optionally, S33 includes: S331, Based on the statistical results of the generated daily language output, obtain the user password information corresponding to the target speaker, generate a random salt value, input the user password information and random salt value into the PBKDF2 key derivation algorithm, and repeatedly calculate through a preset number of iterations to generate an initial key; S332, input the generated initial key into the HKDF key derivation algorithm, and extract and expand it by combining the key expansion salt value and key context information to generate an AES-GCM encryption key; S333: Based on the generated AES-GCM encryption key, a random initialization vector is generated for the generated daily language output statistics. The daily language output statistics, the AES-GCM encryption key, and the random initialization vector are input into the AES-GCM encryption algorithm to obtain encrypted data and integrity authentication information. The encrypted data, integrity authentication information, random initialization vector, and random salt value are associated to generate encrypted daily language output statistics. The encrypted daily language output statistics are stored on the edge device to achieve secure storage of the daily language output statistics.

[0015] The beneficial effects of this invention are: This invention utilizes a camera, microphone array, and edge devices to collaboratively process facial features and lip movement features captured by the camera, along with multi-channel speech signals captured by the microphone array. By combining a spatiotemporal convolutional network and a lightweight Transformer, it achieves precise temporal alignment between lip movement trajectories and speech frames. This enables accurate identification of the target speaker and their corresponding speech start and end times. Even in multi-person dialogues, complex backgrounds, and low-light environments, it can effectively distinguish between the target speaker and non-target speakers, improving the accuracy of speaker detection and identity recognition.

[0016] This invention extracts X-vector voiceprint features using the ECAPA-TDNN model at the edge device, and anonymizes these features by combining X-vector replacement and dynamic noise perturbation techniques. This effectively decouples voiceprint features from user identity information, ensuring that the speech content can be used normally for language output statistics while effectively reducing the risk of voiceprint information leakage and identity reconstruction. Furthermore, by combining the Distil-Whisper speech recognition model, Chinese word segmentation, speaker switching detection, and phoneme-level analysis, this invention comprehensively analyzes multi-dimensional features such as text word count, number of speeches, effective speech duration, pause frequency, and speech rate changes, thereby improving the accuracy and objectivity of daily language output statistics.

[0017] This invention uses the PBKDF2 key derivation algorithm to generate an initial key and the HKDF key derivation algorithm to generate an AES-GCM encryption key. The generated daily language output statistics are then encrypted, and integrity authentication information is generated. The encrypted data is stored on an edge device, achieving local secure storage of the daily language output statistics. This avoids the transmission of raw voice data and voiceprint information to the cloud, ensuring the integrity and confidentiality of statistical data while improving the overall data security protection capability and user privacy protection level of the system. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the counting and voiceprint privacy protection method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the speaker detection and identity recognition process based on visual and speech fusion according to an embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Those skilled in the art may employ other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0021] like Figures 1-2 As shown, a method for automatically counting everyday speech volume and protecting voiceprint privacy based on a camera includes the following steps: S1 collects user facial features and lip movement features in a non-contact manner through a camera, and collects multi-channel speech signals through a microphone array. It uses the camera's lip movement detection and facial feature recognition technology to detect speaking behavior. When a speaker is detected, the microphone array is triggered to collect speech data. The facial features, lip movement features, and multi-channel speech signals are jointly processed. The temporal alignment of the lip trajectory and speech frame is completed through a spatiotemporal convolutional network and a lightweight Transformer. The speaker's identity identifier and speech start and end timestamps are output to obtain the speaker detection and identity recognition results. S2, based on the speaker detection and identity recognition results, performs voiceprint feature stripping and anonymization processing on multi-channel speech signals at the edge device. It uses X-vector replacement and dynamic noise perturbation technology to perturb the voiceprint features in the speech data, decoupling the original voiceprint features from the user identity information and generating anonymized speech data. S3, based on anonymized speech data, uses a speech recognition model to transcribe the anonymized speech. Based on the transcribed text, it counts the total number of words in the text content through Chinese word segmentation, and combines the speaker switching detection results to count the number of times the speaker speaks. At the same time, it estimates the user's daily effective language output based on the phoneme-level duration, pause frequency, and speech rate changes in the anonymized speech data, generating a daily language output statistics result. The daily language output statistics result is encrypted using the AES-GCM algorithm, and a corresponding encryption key is generated through the PBKDF2 and HKDF key derivation algorithms to achieve secure storage of the daily language output statistics result.

[0022] S1 includes: S11: The system continuously captures user facial videos in a non-contact manner using a camera, performs face region detection on the captured video frames, enhances the video frames when the ambient light intensity is lower than a preset light threshold, extracts user facial features within the face region, and extracts lip key points in each video frame using a lip movement detection model. The system calculates the lip movement amplitude based on the positional changes of lip key points between adjacent video frames. When the lip movement amplitude reaches a preset lip movement judgment threshold, the corresponding user is marked as a candidate speaker, and the facial features and lip movement features of the candidate speaker are obtained. S12, when a candidate speaker is detected, the microphone array is triggered to collect multi-channel speech signals. The speech signals of each microphone channel are localized and beamformed to generate a speech signal corresponding to the candidate speaker. Voiceprint features are extracted from the speech signals, and facial features and lip movement features are extracted from the camera acquisition results. The fusion weight of visual features and voiceprint features is dynamically adjusted according to the video occlusion rate and speech noise intensity. The multimodal identity matching value of the candidate speaker is calculated. The candidate speaker whose multimodal identity matching value reaches the preset identity matching threshold is determined as the target speaker. When the user is facing away from or to the side of the camera and cannot obtain effective visual features, the visual feature weight is adjusted to zero, and the target speaker is determined based on the voiceprint features. S13: Encode the lip movement features of the target speaker into a visual temporal feature sequence, and divide the multi-channel speech signal corresponding to the target speaker into continuous speech frames. Extract lip movement trajectory features and speech frame features through a spatiotemporal convolutional network, and then input the lip movement trajectory features and speech frame features into a lightweight Transformer for cross-modal association. Determine the temporal correspondence between each lip movement segment and the corresponding speech frame based on the cross-modal association features, identify the speech start time and speech end time, and bind the target speaker's identity information with the speech start time and speech end time. Output the speaker's identity identifier and speech start and end timestamps to generate speaker detection and identity recognition results.

[0023] S11 includes: S111: The camera continuously captures user facial video at a preset sampling frequency, decomposes the user facial video into video frames arranged in chronological order, and calculates the... The ambient light intensity of each video frame is considered. When the ambient light intensity is lower than a preset light threshold, the Retinex algorithm is used to enhance the video frame. The enhanced video frame or the video frame whose light intensity reaches the preset light threshold is taken as the video frame to be detected, represented as follows: ; ; in, For the first The ambient light intensity of each video frame , These are the width and height of the video frame, respectively. For the first Coordinates in a video frame Pixel brightness value at that location, To enhance the coordinates in the video frame Pixel brightness value at that location, Here is the filtering function used to estimate the illumination components. For convolution operations, To prevent logarithmic operations from involving undefined positive numbers; ; in, To preset the illumination threshold, , These represent the average and standard deviation of the ambient light intensity in historical reference video frames, respectively. This is the illumination adjustment coefficient; S112, the video frame to be detected is input into the face feature extraction model to detect the face position in the video frame, determine the corresponding face region, and extract facial features used to represent user identity within the face region. The face region is then input into the lip movement detection model to locate the lip contour and extract the first... The lip keypoints are set from each video frame and arranged sequentially according to the time order of the video frames to form the user's lip keypoint temporal data. The face feature extraction model uses the YOLO network model, and the lip movement detection model uses the SyncNet network model, represented as follows: ; in, For the first The set of lip key points corresponding to each video frame. For the first The first video frame The coordinates of the location of key points on the lips , The number of key points on the lips; S113, based on the temporal data of lip key points, calculate the positional changes of corresponding lip key points between adjacent video frames, and obtain the first... The lip movement amplitude corresponding to each video frame is used to identify the user in the corresponding video frame as a candidate speaker when the lip movement amplitude reaches a preset lip movement judgment threshold. The facial features extracted from the face region corresponding to the candidate speaker and the changes in the position of lip key points in consecutive video frames are used as the facial features and lip movement features of the candidate speaker, as follows: ; in, For the first The range of lip movements corresponding to each video frame , The first , The first video frame The coordinates of the key points on the lips; ; in, To preset the lip movement detection threshold, , These represent the average and standard deviation of the lip movement amplitude in historical reference video frames, respectively. This is the lip movement adjustment coefficient.

[0024] S12 includes: S121, when a candidate speaker is detected, trigger the... A microphone array consisting of several microphones synchronously acquires multi-channel speech signals, denoted as follows: ,in, Calculate the generalized cross-correlation function between any two microphone channels, determine the inter-channel time delay based on the maximum value of the generalized cross-correlation function, and determine the sound source direction based on the estimated inter-channel time delay and the positional relationship of each microphone in the microphone array. The system performs delay compensation and weighted superposition on the multi-channel speech signals according to the direction of the sound source, maps the direction of the sound source to the position of the candidate speaker in the camera, and generates a speech signal corresponding to the candidate speaker, represented as follows: ; ; ; in, For the first The microphone channel and the first Each microphone channel in candidate delay The generalized cross-correlation value under the following conditions , The first The microphone channel, the first Frequency domain representation of the speech signal from each microphone channel for conjugate, For frequency, The imaginary unit, To prevent positive numbers with a denominator of zero, The estimated time delay between the two microphone channels. For the first The speech signals corresponding to each candidate speaker. For the first Beamforming weights for each microphone channel The direction of the sound source is Time The delay compensation amount corresponding to each microphone channel; S122, extract voiceprint features from the speech signal corresponding to the candidate speaker, and obtain facial features and lip movement features from the camera acquisition results. Use the facial features and lip movement features as the visual features of the candidate speaker, and calculate the visual feature matching value of the candidate speaker respectively. Matching values ​​of voiceprint features The reliability of visual features and the reliability of voiceprint features are calculated based on the video occlusion rate and speech noise intensity of the candidate speakers. The weights of visual features and voiceprint features are then calculated based on these reliability values, expressed as follows: ; ; ; ; ; ; ; ; ; in, To extract the first from the current camera capture results Visual feature vectors of candidate speakers For the first Each candidate speaker corresponds to a pre-registered visual feature template vector. This is the voiceprint feature vector extracted from the current speech signal corresponding to the candidate speaker. For the first Each candidate speaker corresponds to a pre-registered voiceprint feature template vector. For the first Reliability of visual features of candidate speakers For the first Video obscuring rate of each candidate speaker For the first Reliability of voiceprint features of candidate speakers The effective speech intensity of the speech signal corresponding to the candidate speaker. The noise intensity in the speech signal. For the first Visual feature weights of each candidate speaker For the first The voiceprint feature weights of each candidate speaker For the first The occlusion area in the face region of each candidate speaker. For the first Total area of ​​the face region of each candidate speaker This represents the number of sampling points contained in the speech segment corresponding to the candidate speaker. For the first The candidate speaker in the first The speech amplitude at each sampling point This represents the number of sampling points in the noise segment. For the first The noise amplitude corresponding to each noise sampling point; S123, based on visual feature matching values, voiceprint feature matching values, visual feature weights, and voiceprint feature weights, calculate the... The multimodal identity matching value of the candidate speaker is used to determine the next step. When the multimodal identity matching value reaches a preset identity matching threshold, the candidate speaker will be... When a candidate speaker is identified as the target speaker, and their facial features or lip movements cannot be effectively extracted due to their back or side facing the camera, the visual feature weight is set to 0, and the voiceprint feature weight is set to 1, as follows: ; in, For the first Multimodal identity matching values ​​of each candidate speaker; ; in, To preset the identity matching threshold, , These represent the average and standard deviation of historical reference identity matching values, respectively. This is the identity matching adjustment coefficient.

[0025] S13 includes: S131, Arrange the key lip points and their positional changes of the target speaker in consecutive video frames according to the time sequence of the video frames, and then... The lip movement features corresponding to each video frame are encoded into a visual feature vector. This forms a visual temporal feature sequence. Simultaneously, the speech signal corresponding to the target speaker is divided into continuous speech frames according to a preset frame length and frame shift. Each speech frame is represented as This forms a sequence of speech frames, represented as: ; ; in, For the visual temporal feature sequence of the target speaker, For the target speaker's speech frame sequence, The number of video frames. This refers to the number of audio frames. S132, the visual temporal feature sequence and the speech frame sequence are input into a spatiotemporal convolutional network to extract the spatial morphological changes and temporal evolution of continuous lip movements, obtaining the lip movement trajectory feature sequence. The temporal-frequency changes of continuous speech frames are also extracted to obtain the speech frame feature sequence. The lip movement trajectory feature sequence is used as the query information, and the speech frame feature sequence is used as the key and value input to a lightweight Transformer to calculate the cross-modal attention matrix, expressed as: ; ; ; ; ; ; in, This is a sequence of lip movement trajectory features. This is the spatiotemporal convolutional network mapping function used to extract lip movement trajectory features. Pooling operations are performed along the spatial height and spatial width directions. It is a non-linear activation function. These are the 3D convolution kernel parameters for the first layer of the visual spatiotemporal convolutional network. For the first 3D convolution kernel parameters of a layered visual spatiotemporal convolutional network. For three-dimensional convolution operations, Input the feature tensor into the visual spatiotemporal convolutional network. These are the bias parameters for the first layer of the visual spatiotemporal convolutional network. For the first Bias parameters of layered visual spatiotemporal convolutional networks For speech frame feature sequences, This is the spatiotemporal convolutional network mapping function used to extract speech frame features. Pooling operations performed along the frequency dimension These are the two-dimensional convolution kernel parameters for the first-layer speech time-frequency convolutional network. For the first The two-dimensional convolution kernel parameters of a layered speech time-frequency convolutional network. This is a two-dimensional convolution operation. Input the feature tensor to the speech time-frequency convolutional network. These are the bias parameters for the first layer of the speech time-frequency convolutional network. For the first Bias parameters of a multilayer speech time-frequency convolutional network. This is the cross-modal correlation matrix between lip movement segments and speech frames. This is a query matrix generated from lip movement trajectory features. The key matrix is ​​generated from the features of the speech frames. For cross-modal association features, , , These are the query mapping matrix, the key mapping matrix, and the value mapping matrix, respectively. The feature dimension of the key vector. This is the matrix transpose. S133, based on the cross-modal correlation matrix, for the first For each lip movement segment, the speech frame with the highest correlation value is selected as the corresponding speech frame. The time corresponding to the first speech frame that consecutively satisfies the correlation condition is determined as the speech start time, and the time corresponding to the last speech frame that consecutively satisfies the correlation condition is determined as the speech end time. The target speaker's identity information is bound to the speech start time and speech end time respectively, and the speaker identity identifier and speech start and end timestamps are output to generate the speaker detection and identity recognition result, represented as: ; ; ; in, In order to be with the first The sequence number of the speech frame corresponding to each lip movement segment. For the first The mouth movement segment and the first Cross-modal correlation values ​​between speech frames The moment the voice begins. The moment the voice message ends. This is the sequence number of the first corresponding speech frame. This is the sequence number of the last corresponding audio frame. For speech frame shifting, The length of the audio frame.

[0026] S2 includes: S21, Based on the speaker detection and identity recognition results, extract the multi-channel speech signal corresponding to the target speaker at the edge device. After preprocessing the multi-channel speech signal, input it into the ECAPA-TDNN model to extract the X-vector voiceprint feature vector of the target speaker, and calculate the cosine distance between the X-vector voiceprint feature vector and each pseudo-X-vector feature vector in the pre-trained X-vector library. ,when When the time comes, the corresponding pseudo-X-vector is determined as the target replacement vector, represented as: ; ; in, For the first X-vector of the target speaker's voiceprint feature vector To input into the ECAPA-TDNN model, For the first The speech signal corresponding to the target speaker. For the pre-trained X-vector library A pseudo-X-vector feature vector; The ECAPA-TDNN model specifically includes: (1) The first The speech signal of the target speaker The speech is divided into consecutive speech frames, and each speech frame undergoes frequency domain transformation and filtering to form an acoustic feature sequence, represented as follows: ; in, For the first A sequence of acoustic features of a target speaker For the first Acoustic feature vectors of each speech frame This represents the total number of audio frames. (2) Input the acoustic feature sequence into the frame-level feature coding network of ECAPA-TDNN to obtain the deep speaker features corresponding to each speech frame, as follows: ; ; in, After the first layer of time-delay convolution processing, the th Feature vectors of each speech frame For the first The frame-level feature vectors output by each feature coding layer. , These are the weight parameters and bias parameters of the first convolutional layer, respectively. This is a one-dimensional time-delay convolution operation. For the first Multi-scale residual feature extraction function of the layer For the first The channel attention weighting function of the layer, This represents the total number of frame-level feature coding layers in ECAPA-TDNN; The outputs of each coding layer are aggregated to obtain the first... Aggregated speaker characteristics of individual speech frames: ; in, For the first Aggregated voiceprint features of each speech frame For feature concatenation operations; (3) Calculate the attention weights based on the aggregated speaker features of each speech frame: ; in, For the first Attention weights for each speech frame. , These are the attention mapping matrix and the bias vector, respectively. For attention query vectors, Indexing for voice frames; Calculate the weighted mean vector and weighted standard deviation vector based on attention weights: ; ; in, For the first Weighted mean features of each target speaker For weighted standard deviation characteristics, Square each element of the vector; The weighted mean vector and the weighted standard deviation vector are concatenated, and then the X-vector voiceprint feature vector is generated through fully connected mapping and normalization, as follows: ; in, , These are the weight matrix and bias vector of the voiceprint vector mapping layer, respectively; S22, based on the first The X-vector voiceprint feature vector of each target speaker is selected to satisfy... The pseudo-X-vector feature vector is used as the target replacement vector to replace the X-vector voiceprint feature vector, generating an anonymized voiceprint feature vector. The voiceprint feature replacement process is performed on the multi-channel speech signal corresponding to the target speaker based on the anonymized voiceprint feature vector, so that the processed speech signal retains the speech content information and reduces the voiceprint feature information related to the target speaker's identity in the multi-channel speech signal, thereby generating the initial anonymized speech data. S23. For the generated initial anonymized speech data, analyze its speech spectrum features, determine the perturbation weights corresponding to different spectral regions based on the sensitivity of voiceprint features, apply dynamic noise perturbation to the speech spectrum features, and convert the perturbed speech spectrum features into a time-domain speech signal to obtain the anonymized speech data, represented as: ; ; in, For the perturbation of the first The speech spectrum characteristics of a target speaker The speech spectral features are derived from the initial anonymized speech data. Spectral position The corresponding dynamic perturbation weights, Here is the noise perturbation matrix. For the first The initial anonymized speech time-domain signal of the target speaker. The window length for the short-time Fourier transform. Assign time frame numbers, The window movement step size, For STFT window functions, These are the basis functions for the Fourier transform.

[0027] S3 includes: S31, Based on the generated anonymized speech data, the speech content of the anonymized speech data is transcribed using a speech recognition model to generate speech-transcribed text, and a text word set is obtained through Chinese word segmentation. The number of words in the text word set is counted to generate text word count features. At the same time, the anonymized speech data is divided into speech segments. S32, based on speech-to-text, text word count features, and anonymized speech data, uses speaker switching detection technology to identify speaker switching positions in continuous speech segments and counts the number of times the target speaker speaks. At the same time, it performs phoneme-level analysis on the anonymized speech data, extracts phoneme-level duration, pause frequency, and speech rate variation features, and calculates the target speaker's daily language output by combining text word count, number of times spoken, effective language duration, pause frequency, and speech rate variation features, generating daily language output statistics results; S33. Based on the generated daily language output statistics, an initial key is generated by combining user password information and random salt value with the PBKDF2 key derivation algorithm. The initial key is then expanded using the HKDF key derivation algorithm to generate an AES-GCM encryption key. The daily language output statistics are then encrypted using the AES-GCM encryption key to generate encrypted daily language output statistics including encrypted data and integrity authentication information, thus achieving secure storage of the daily language output statistics.

[0028] S31 includes: S311, the first Anonymized speech data from a target speaker is input into a speech recognition model to identify the speech content in the speech signal and obtain the corresponding speech-to-text. The speech recognition model uses the lightweight Distil-Whisper speech recognition model, which specifically includes: (1) The anonymized speech data is segmented into frames, and the corresponding acoustic features are extracted to generate an acoustic feature sequence. The acoustic feature sequence is then input into the Distil-Whisper encoder to obtain the speech hidden feature representation. The Distil-Whisper encoder is represented as follows: ; ; ; ; ; ; ; ; in, For the first The target speaker in the first The initial coding features corresponding to each speech frame For the first The acoustic feature vector corresponding to each speech frame The linear mapping matrix of acoustic features. For linear mapping bias parameters, For the first Position encoding of each audio frame. Input the feature sequence to the Distil-Whisper encoder. For the first Layer Transformer encoder output features Number the Transformer encoding layer. For the first Layer attention output features, For querying the matrix, The key matrix, For value matrices, , , The first The linear mapping matrix corresponding to the layer, For the attention key vector dimension, It is a feedforward neural network. For layer normalization function, For speech hidden feature representation, The number of Transformer layers for the Distil-Whisper encoder; (2) Input the speech hidden feature representation into the Distil-Whisper decoder, and predict the probability of the next text unit step by step based on the generated text sequence, as follows: ; in, To achieve the desired result under the constraints of existing text sequences and speech hiding features, the first... The predicted probability of each text unit. For the first One text unit to be predicted For the first The text sequence that was generated before the text unit For decoder number The hidden state of the step output, Output a mapping matrix for the text. For output bias parameters; The text unit with the highest predicted probability is selected based on the predicted probability of the text unit, which is expressed as: ; in, For the first The text unit determined step by step; Repeat the above process until a complete text sequence is generated, represented as: ; in, For the first A candidate text sequence for a target speaker. This represents the number of text units. (3) Based on the candidate text sequence, streaming decoding is used to concatenate and semantically correct the continuously generated text units to obtain the final speech-to-text, which is represented as: ; in, For the first The speech-to-text transcription of the target speaker This is the Distil-Whisper decoding function; To improve the reliability of transcribed text, the text confidence score is calculated based on the average predicted probability corresponding to the text sequence, and is expressed as: ; in, For the first Text transcription confidence of a target speaker For the first The predicted probability of each text unit; when When the time comes, the corresponding speech-to-text will be output, where, Set a preset text confidence threshold; ; in, , These represent the mean and standard deviation of the confidence scores for historical text transcription, respectively. This is the text confidence adjustment coefficient; S312, based on speech-to-text transcription, uses a Chinese word segmentation tool to segment the speech-to-text into words, converting continuous text into multiple independent word units, generating a text word set, and statistically analyzing the text word count features based on the text word set, represented as: ; ; in, For the first A set of words in the text of a target speaker. For the first The first target speaker's Each word unit The number of words in the text word set. For the first Text word count features of a target speaker The number of elements in the text word set. The number of words in the text word set; S313, based on the corresponding anonymized speech data, the continuous speech signal is segmented according to the speech activity detection results. Regions with continuous speech activity are divided into independent speech segments, generating a speech segment set. The time interval between adjacent speech segments is calculated based on their temporal continuity. If the time interval is not greater than a preset speech interval threshold, it is retained as an independent speech segment; otherwise, the speech segments are merged. The final speech segment set is represented as follows: ; ; ; in, For the first A collection of speech fragments from a target speaker. For the first The first target speaker's A voice clip, The number of speech segments obtained by dividing anonymized speech data. For the first The first audio segment and the first The time interval between speech segments For the first The end time of each audio segment For the first The start time of each audio segment To preset the speech interval threshold, , These represent the average and standard deviation of historical speech pause times, respectively. This is the pause interval adjustment coefficient.

[0029] S32 includes: S321, Based on the generated set of speech segments, extract the semantic and acoustic features of adjacent speech segments respectively, and calculate the speaker switching determination value according to the semantic and acoustic differences between adjacent speech segments. ,when At that time, the first The first audio segment and the first The position between each audio segment is used to determine the speaker switching position. To preset the speaker switching threshold, and to count the number of times the target speaker speaks based on the identified speaker switching location, it is represented as follows: ; ; ; ; in, The semantic differences between adjacent speech segments The acoustic differences between adjacent speech segments , These are the semantic feature vectors of the transcribed text corresponding to adjacent speech segments. , These are the acoustic feature vectors of adjacent speech segments. , These are semantic difference weights and acoustic difference weights, respectively. To prevent positive numbers with a denominator of zero, For the first Number of times each target speaker speaks. This is an indicator function that takes the value 1 if the condition is true, and 0 otherwise. ; in, , These are the average and standard deviation of the historical speaker switching judgment values, respectively. Switch the adjustment coefficient for the speaker; S322: Perform phoneme-level alignment on the anonymized speech data, identify the start and end times of each valid phoneme, calculate the duration of each valid phoneme, sum the durations of all valid phonemes to obtain the effective speech duration, calculate the pause frequency based on the number of pause segments in the anonymized speech data whose duration reaches a preset pause duration threshold, calculate the speech rate based on the number of words in each consecutive speech segment and the effective speech duration, and calculate the speech rate variation characteristics between adjacent speech segments, expressed as: ; ; ; ; ; in, For the first The first target speaker's The duration of each effective phoneme , These are the start and end times of the effective phoneme, respectively. For the first Effective speech duration for each target speaker For the number of effective phonemes, For the first The frequency of pauses of the speaker in the target audience. This is to determine the number of pause segments whose duration reaches a preset pause duration threshold within a statistical period. The duration of speech observation for the target speaker within the statistical period. For the first The first target speaker's The speech rate of each audio segment For the first The number of words corresponding to each audio segment For the first Effective speech duration of each speech segment Characteristics of speech rate variation; S323, input text word count features, number of utterances, effective language duration, pause frequency, and speech rate variation features into the language quantity statistical model, and calculate the first... The daily language output of a target speaker is calculated, and the speaker's identity, daily language output, and statistical time are correlated to generate a statistical result of daily language output, which is represented as follows: ; ; in, For the first The daily language output of the target speaker , , , , These are the weighting coefficients corresponding to the text word count, number of utterances, effective language duration, pause frequency, and speech rate variation characteristics. For the first Statistical results of the daily language output of the target speaker As a means of identifying the speaker, This corresponds to the statistical date or statistical period.

[0030] S33 includes: S331, based on the statistical results of the generated daily language output, obtain the user password information corresponding to the target speaker, and generate a random salt value. Input the user password information and the random salt value into the PBKDF2 key derivation algorithm, and repeatedly calculate through a preset number of iterations to generate the initial key. The preset number of iterations is set to 50,000 to 100,000 times, as shown below: ; in, For the first The initial key corresponding to each target speaker. For the first The target speaker's user password information. For the first A random salt value generated for each target speaker. This is the preset number of iterations for the PBKDF2 key derivation algorithm. The output length of the initial key. Derived functions for the PBKDF2 key; S332, input the generated initial key into the HKDF key derivation algorithm, and extract and expand it by combining the key expansion salt value and key context information to generate the AES-GCM encryption key, represented as: ; in, For the first The AES-GCM encryption key corresponding to each target speaker. To extend the salt value of the HKDF key, This refers to key context information, which is used to distinguish different target speakers, devices, or data purposes. The target length of the AES-GCM encryption key. Derivation functions for HKDF keys; S333, based on the generated AES-GCM encryption key, generates a random initialization vector for the generated daily language output statistics. The daily language output statistics, the AES-GCM encryption key, and the random initialization vector are then input into the AES-GCM encryption algorithm to obtain encrypted data and integrity authentication information. The encrypted data, integrity authentication information, random initialization vector, and random salt value are correlated to generate encrypted daily language output statistics. These encrypted statistics are then stored on an edge device, achieving secure storage of the daily language output statistics, as shown below: ; ; in, For the first Encrypted data corresponding to the statistical results of the daily language output of a target speaker. For the integrity of authentication information, To initialize a vector randomly, Additional authentication data that does not require encryption but participates in integrity verification. For AES-GCM encryption functions, For the first Statistical data on the encrypted daily language output of a target speaker.

[0031] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0032] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatic counting of everyday speech volume and voiceprint privacy protection based on a camera, characterized in that, Includes the following steps: S1: The system collects user facial features and lip movement features in a non-contact manner through a camera, and collects multi-channel speech signals through a microphone array. It uses lip movement detection and facial feature recognition technology of the camera to detect speaking behavior. When a speaker is detected, the microphone array is triggered to collect speech data. The facial features, lip movement features and multi-channel speech signals are jointly processed. The lip trajectory and speech frame are aligned in time through a spatiotemporal convolutional network and a lightweight Transformer. The speaker's identity identifier and speech start and end timestamps are output to obtain the speaker detection and identity recognition results. S2, based on the speaker detection and identity recognition results, the multi-channel speech signal is subjected to voiceprint feature stripping and anonymization processing at the edge device end. X-vector replacement and dynamic noise perturbation technology are used to perturb the voiceprint features in the speech data to generate anonymized speech data. S3, based on anonymized speech data, uses a speech recognition model to transcribe the anonymized speech. Based on the transcribed text, it counts the total number of words in the text content through Chinese word segmentation, and combines the speaker switching detection results to count the number of times the speaker speaks. At the same time, it estimates the user's daily effective language output based on the phoneme-level duration, pause frequency, and speech rate changes in the anonymized speech data, and generates a daily language output statistics result. The daily language output statistics result is encrypted using the AES-GCM algorithm, and the corresponding encryption key is generated through the PBKDF2 and HKDF key derivation algorithms.

2. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 1, characterized in that, S1 includes: S11, continuously capture user facial videos in a non-contact manner using a camera, perform face region detection on the captured video frames, enhance the video frames when the ambient light intensity is lower than a preset light threshold, extract user facial features within the face region, and extract lip key points in each video frame using a lip movement detection model. Calculate the lip movement amplitude based on the positional changes of lip key points between adjacent video frames. When the lip movement amplitude reaches a preset lip movement judgment threshold, mark the corresponding user as a candidate speaker and obtain the candidate speaker's facial features and lip movement features. S12, when a candidate speaker is detected, the microphone array is triggered to collect multi-channel speech signals. The speech signals of each microphone channel are localized and beamformed to generate a speech signal corresponding to the candidate speaker. Voiceprint features are extracted from the speech signals, and facial features and lip movement features are extracted from the camera acquisition results. The fusion weight of visual features and voiceprint features is dynamically adjusted according to the video occlusion rate and speech noise intensity. The multimodal identity matching value of the candidate speaker is calculated. The candidate speaker whose multimodal identity matching value reaches the preset identity matching threshold is determined as the target speaker. When the user is facing away from or to the side of the camera and cannot obtain effective visual features, the visual feature weight is adjusted to zero, and the target speaker is determined based on the voiceprint features. S13, the lip movement features of the target speaker are encoded into a visual temporal feature sequence, and the multi-channel speech signal corresponding to the target speaker is divided into continuous speech frames. The lip movement trajectory features and speech frame features are extracted by a spatiotemporal convolutional network. The lip movement trajectory features and speech frame features are then input into a lightweight Transformer for cross-modal association. The temporal correspondence between each lip movement segment and the corresponding speech frame is determined based on the cross-modal association features. The start time and end time of the speech are identified, and the identity information of the target speaker is bound to the start time and end time of the speech. The speaker identity identifier and the start and end timestamps of the speech are output, and the speaker detection and identity recognition results are generated.

3. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 2, characterized in that, S11 includes: S111, continuously capture user facial video using a camera at a preset sampling frequency, decompose the user facial video into video frames arranged in chronological order, and calculate the first... The ambient light intensity of each video frame is used. When the ambient light intensity is lower than the preset light threshold, the Retinex algorithm is used to enhance the video frame, and the enhanced video frame or the video frame whose light intensity reaches the preset light threshold is used as the video frame to be detected. S112, the video frame to be detected is input into the face feature extraction model to detect the face position in the video frame, determine the corresponding face region, and extract facial features used to represent user identity within the face region. The face region is then input into the lip movement detection model to locate the lip contour and extract the first... The set of lip key points in each video frame is arranged continuously according to the time sequence of the video frames to form the user's lip key point time sequence data. S113, based on the temporal data of lip key points, calculate the positional changes of corresponding lip key points between adjacent video frames, and obtain the first... The lip movement amplitude corresponding to each video frame is used to mark the user in the corresponding video frame as a candidate speaker when the lip movement amplitude reaches the preset lip movement judgment threshold. The facial features extracted from the face area corresponding to the candidate speaker and the changes in the position of the key points of the lips in the continuous video frames are used as the facial features and lip movement features of the candidate speaker.

4. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 3, characterized in that, S12 includes: S121, when a candidate speaker is detected, trigger the... A microphone array consisting of several microphones synchronously acquires multi-channel speech signals, denoted as follows: ,in, Calculate the generalized cross-correlation function between any two microphone channels, determine the inter-channel time delay based on the maximum value of the generalized cross-correlation function, and determine the sound source direction based on the estimated inter-channel time delay and the positional relationship of each microphone in the microphone array. The system performs delay compensation and weighted superposition on the multi-channel speech signals according to the direction of the sound source, and maps the direction of the sound source to the position of the candidate speaker in the camera to generate a speech signal corresponding to the candidate speaker. S122, extract voiceprint features from the speech signal corresponding to the candidate speaker, and obtain facial features and lip movement features from the camera acquisition results. Use the facial features and lip movement features as the visual features of the candidate speaker, and calculate the visual feature matching value of the candidate speaker respectively. Matching values ​​of voiceprint features The reliability of visual features and the reliability of voiceprint features are calculated based on the video occlusion rate and speech noise intensity of the candidate speaker. The weights of visual features and voiceprint features are calculated based on the reliability of visual features and the reliability of voiceprint features. S123, based on visual feature matching values, voiceprint feature matching values, visual feature weights, and voiceprint feature weights, calculate the... The multimodal identity matching value of the candidate speaker is used to determine the next step. When the multimodal identity matching value reaches a preset identity matching threshold, the candidate speaker will be... When a candidate speaker is identified as the target speaker, and the candidate speaker is facing away from or to the side of the camera, making it impossible to effectively extract their facial features or lip movement features, the visual feature weight is set to 0 and the voiceprint feature weight is set to 1.

5. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 4, characterized in that, S13 includes: S131, Arrange the key lip points and their positional changes of the target speaker in consecutive video frames according to the time sequence of the video frames, and then... The lip movement features corresponding to each video frame are encoded into a visual feature vector. This forms a visual temporal feature sequence. Simultaneously, the speech signal corresponding to the target speaker is divided into continuous speech frames according to a preset frame length and frame shift. Each speech frame is represented as This forms a sequence of speech frames; S132, the visual temporal feature sequence and the speech frame sequence are respectively input into the spatiotemporal convolutional network to extract the spatial morphological changes and temporal evolution of continuous mouth movements to obtain the lip movement trajectory feature sequence, and the temporal frequency changes of continuous speech frames are extracted to obtain the speech frame feature sequence. The lip movement trajectory feature sequence is used as query information, and the speech frame feature sequence is used as key and value input into the lightweight Transformer to calculate the cross-modal attention matrix. S133, based on the cross-modal correlation matrix, for the first For each lip movement segment, the speech frame with the highest correlation value is selected as the corresponding speech frame. The time corresponding to the first speech frame that continuously meets the correlation condition is determined as the speech start time, and the time corresponding to the last speech frame that continuously meets the correlation condition is determined as the speech end time. The identity information of the target speaker is bound to the speech start time and the speech end time respectively, and the speaker identity identifier and speech start and end timestamps are output to generate speaker detection and identity recognition results.

6. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 5, characterized in that, S2 includes: S21, based on the speaker detection and identity recognition results, extract the multi-channel speech signal corresponding to the target speaker at the edge device. After preprocessing the multi-channel speech signal, input it into the ECAPA-TDNN model to extract the X-vector voiceprint feature vector of the target speaker, and calculate the cosine distance between the X-vector voiceprint feature vector and each pseudo-X-vector feature vector in the pre-trained X-vector library. ,when When the time comes, the corresponding pseudo-X-vector will be determined as the target replacement vector; S22, based on the first The X-vector voiceprint feature vector of each target speaker is selected to satisfy... The pseudo-X-vector feature vector is used as the target replacement vector to replace the X-vector voiceprint feature vector, generating an anonymized voiceprint feature vector. And based on the anonymized voiceprint feature vector, the multi-channel speech signal corresponding to the target speaker is processed by voiceprint feature replacement to generate initial anonymized speech data; S23. For the generated initial anonymized speech data, analyze its speech spectrum features, determine the perturbation weights corresponding to different spectrum regions based on the sensitivity of voiceprint features, perform dynamic noise perturbation on the speech spectrum features, and convert the perturbed speech spectrum features into a time-domain speech signal to obtain anonymized speech data.

7. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 6, characterized in that, S3 includes: S31, based on the generated anonymized speech data, the speech content of the anonymized speech data is transcribed using a speech recognition model to generate speech-transcribed text, and a text word set is obtained through Chinese word segmentation. The number of words in the text word set is counted to generate text word count features. At the same time, the anonymized speech data is divided into speech segments. S32, based on speech-to-text, text word count features, and anonymized speech data, speaker switching detection technology is used to identify speaker switching positions in continuous speech segments and count the number of times the target speaker speaks. At the same time, phoneme-level analysis is performed on the anonymized speech data to extract phoneme-level duration, pause frequency, and speech rate change features. Combined with text word count, number of times spoken, effective language duration, pause frequency, and speech rate change features, the daily language output of the target speaker is calculated, and a daily language output statistical result is generated. S33. Based on the generated daily language output statistics, an initial key is generated by combining user password information and random salt value with the PBKDF2 key derivation algorithm, and the initial key is expanded using the HKDF key derivation algorithm to generate an AES-GCM encryption key. The daily language output statistics are then encrypted using the AES-GCM encryption key to generate encrypted daily language output statistics including encrypted data and integrity authentication information, thereby achieving secure storage of the daily language output statistics.

8. The method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera according to claim 7, characterized in that, S31 includes: S311, the first Anonymous speech data of a target speaker is input into a speech recognition model to identify the speech content in the speech signal and obtain the corresponding speech-to-text. The speech recognition model adopts the lightweight speech recognition model Distil-Whisper. S312, Based on the speech-to-text transcription, the speech-to-text is segmented into words using a Chinese word segmentation tool, the continuous text is converted into multiple independent word units, a text word set is generated, and the text word count feature is statistically analyzed based on the text word set; S313, based on the corresponding anonymized speech data, the continuous speech signal is segmented according to the speech activity detection results, the region with continuous speech activity is divided into independent speech segments, a speech segment set is generated, and the time interval is calculated according to the temporal continuity relationship between adjacent speech segments. When the time interval is not greater than the preset speech interval threshold, it is retained as an independent speech segment; otherwise, the speech segments are merged to finally generate a speech segment set.

9. A method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera, as described in claim 8, is characterized in that... S32 includes: S321, Based on the generated set of speech segments, extract the semantic and acoustic features of adjacent speech segments respectively, and calculate the speaker switching determination value according to the semantic and acoustic differences between adjacent speech segments. ,when At that time, the first The first audio segment and the first The position between each audio segment is used to determine the speaker switching position. A preset speaker switching threshold is set, and the number of times the target speaker speaks is counted based on the identified speaker switching position; S322, perform phoneme-level alignment on the anonymized speech data, identify the start and end times of each effective phoneme, calculate the duration of each effective phoneme, accumulate the durations of all effective phonemes to obtain the effective speech duration, calculate the pause frequency based on the number of pause segments in the anonymized speech data whose duration reaches the preset pause duration threshold, calculate the speech rate based on the number of words in each continuous speech segment and the effective speech duration, and calculate the speech rate change characteristics between adjacent speech segments; S323, input text word count features, number of utterances, effective language duration, pause frequency, and speech rate variation features into the language quantity statistical model, and calculate the first... The daily language output of a target speaker is calculated, and the target speaker's identity, daily language output, and statistical time are linked to generate statistical results of daily language output.

10. A method for automatic counting of daily speech volume and voiceprint privacy protection based on a camera, as described in claim 9, is characterized in that... S33 includes: S331, Based on the statistical results of the generated daily language output, obtain the user password information corresponding to the target speaker, generate a random salt value, input the user password information and random salt value into the PBKDF2 key derivation algorithm, and repeatedly calculate through a preset number of iterations to generate an initial key; S332, input the generated initial key into the HKDF key derivation algorithm, and extract and expand it by combining the key expansion salt value and key context information to generate an AES-GCM encryption key; S333: Based on the generated AES-GCM encryption key, a random initialization vector is generated for the generated daily language output statistics. The daily language output statistics, the AES-GCM encryption key, and the random initialization vector are input into the AES-GCM encryption algorithm to obtain encrypted data and integrity authentication information. The encrypted data, integrity authentication information, random initialization vector, and random salt value are associated to generate encrypted daily language output statistics. The encrypted daily language output statistics are stored on the edge device to achieve secure storage of the daily language output statistics.

Images