A method of dictation shorthand

Abstract

The application provides a method for dictation, and relates to the technical field of voice processing, and the method comprises the following steps: when there is valid voice, performing voiceprint feature detection on the valid voice to obtain voiceprint feature data; performing identity recognition on the voiceprint feature data to determine the speaker in the valid voice; generating an interaction prompt for a non-pre-authorized speaker to obtain the dictation authorization of the speaker; when the dictation authorization of all speakers in the valid voice has been obtained, performing dictation on the voice of all speakers; when the dictation authorization of all speakers has not been obtained, retaining the audio of the authorized speakers according to the voiceprint feature data and performing the dictation to obtain the authorized dictation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, in particular to a call shorthand method. BACKGROUND

[0002] Call shorthand is a speech processing function that automatically converts speech content in a call into a text record in real time or afterwards, extracts key information, and helps users quickly obtain the core content of the call. Its core goal is to improve information retention efficiency, reduce manual recording burden, and support efficient communication needs in multiple scenarios.

[0003] In related technologies, call shorthand technology can achieve real-time speech recognition and shorthand, but it raises privacy issues, as it defaults to recording both parties' call content and cannot dynamically control the data recording range. During the call, there is a contradiction between call real-time and authorization mechanism lag. SUMMARY

[0004] The problem solved by the present application is how to perform call shorthand after authorization.

[0005] To solve the above problems, the present application provides a call shorthand method.

[0006] In a first aspect, the present application provides a call shorthand method, comprising:

[0007] When there is valid speech, perform voiceprint feature detection on the valid speech to obtain voiceprint feature data;

[0008] Perform identity recognition on the voiceprint feature data to determine the speaker in the valid speech;

[0009] Generate an interaction prompt to the un-pre-authorized speaker to obtain the shorthand authorization of the speaker;

[0010] When the shorthand authorization of all speakers in the valid speech has been obtained, perform call shorthand on the speech of all speakers;

[0011] When the shorthand authorization of all speakers has not been obtained, retain the audio of the authorized speakers according to the voiceprint feature data and perform the call shorthand.

[0012] Optionally, when the shorthand authorization of all speakers in the valid speech has been obtained, performing call shorthand on the speech of all speakers comprises:

[0013] Encrypting the speech of all speakers to obtain a merged speech stream through a time domain interleaving algorithm;

[0014] Performing the call shorthand on the merged speech stream.

[0015] Optionally, after the speech of the all speakers is encrypted and the merged speech stream is obtained through the time domain interleaving algorithm, the method further comprises:

[0016] Every preset time step, the quality loss of the merged speech stream is monitored.

[0017] When the quality loss is greater than a preset threshold, the shorthand authorization of the all speakers is triggered again.

[0018] Optionally, the every preset time step, the quality loss of the merged speech stream is monitored comprises:

[0019] The merged speech stream is monitored for word error rate and character error rate.

[0020] Optionally, before the voiceprint feature detection on the valid speech is performed to obtain voiceprint feature data when the valid speech exists, the method further comprises:

[0021] The call speech is acquired, and the speech stream meeting the valid speech condition is taken as the valid speech, wherein the valid speech condition comprises at least one of the sound duration exceeding a preset duration, the sound frequency meeting a preset fundamental frequency range, and the sound harmonic structure meeting a preset harmonic structure.

[0022] Optionally, before the voiceprint feature detection on the valid speech is performed to obtain voiceprint feature data when the valid speech exists, the method further comprises:

[0023] The valid speech is subjected to echo cancellation and noise suppression, and is divided according to a preset window length to obtain speech segments;

[0024] The speech segments are subjected to first-order high-pass filtering to obtain pre-emphasis speech;

[0025] The number of the pre-emphasis speech is taken as the sample number, the number of channels of the pre-emphasis speech is taken as the channel number, the frequency dimension and the time dimension of the pre-emphasis speech are taken as the height and the width respectively, and the pre-emphasis speech is sent into a convolutional neural network to obtain a silence detection threshold;

[0026] According to the silence detection threshold, the pre-emphasis speech is subjected to silence detection to eliminate silence segments to obtain a silence detection result;

[0027] The pre-emphasis speech is subjected to zero-crossing rate detection to obtain a zero-crossing rate detection result;

[0028] According to the silence detection result and the zero-crossing rate detection result, a voice activity detection decision result is obtained.

[0029] Optionally, the voiceprint feature detection on the valid voice to obtain voiceprint feature data comprises:

[0030] Constructing a multi-channel triangular filter;

[0031] Extracting a feature vector of the valid voice according to the multi-channel triangular filter, wherein the feature vector comprises a Mel frequency cepstrum coefficient, a first-order differential Mel frequency cepstrum coefficient and an energy feature;

[0032] Optimizing the dimension of the feature vector based on a supervised dimension reduction algorithm to obtain the voiceprint feature data.

[0033] Optionally, the voiceprint feature data is subjected to identity recognition to determine the speaker in the valid voice, comprising:

[0034] Dividing a voiceprint segment based on a voiceprint change point, wherein the voiceprint change point comprises a mute point and a sound energy mutation point;

[0035] By a clustering algorithm, similar voiceprint features are classified as the same speaker in the voiceprint feature data corresponding to the voiceprint segment.

[0036] Optionally, the interaction prompt comprises at least one of a visual interaction interface or a text prompt;

[0037] After the interaction prompt is generated for the speaker who is not pre-authorized, further comprising:

[0038] In response to a shorthand authorization instruction, an encrypted channel is established, and the shorthand authorization state of the speaker is changed according to the shorthand authorization instruction.

[0039] Optionally, when the shorthand authorization of the speaker is not obtained, the audio of the speaker who is authorized is retained according to the voiceprint feature data, and the call shorthand is performed, comprising:

[0040] Obtaining authorized voiceprint feature data of the speaker who is authorized;

[0041] The audio corresponding to the authorized voiceprint feature data greater than a preset likelihood ratio is taken as target audio;

[0042] The call shorthand is performed on the audio corresponding to the speaker who is authorized.

[0043] The call shorthand method has the following advantages:

[0044] Since the voiceprint feature has individual uniqueness, different speaker identities can be effectively distinguished, and the recognition stability can be maintained in different contexts. After extracting the voiceprint feature data from the effective speech, the voiceprint database stored in advance can be matched based on the data, so as to determine the identity of the current speaker. On this basis, the recognized identity information is associated with the authorized state, which provides a basis for subsequent verification whether to allow shorthand. The unpre-authorized speaker is actively sent an authorization request, which can not only guarantee the right to know of the user, but also take into account the real-time and privacy protection needs of the conversation. When all the speakers participating in the conversation are authorized, the conversation shorthand text can be generated by shorthand for all the speech contents; if there is an un-authorized speaker, the voiceprint feature data is used to dynamically filter the speech segments of the un-authorized speaker, and only the audio and corresponding text of the authorized part are retained, so as to avoid the leakage of sensitive information. Through the combination of voiceprint recognition and dynamic authorization mechanism, the strong demand of the user for privacy protection is met, and the efficient recording and arrangement of authorized content are guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the conversation shorthand method of the embodiment of the present application is shown in the figure.

[0046] Figure 2 The flowchart of the conversation shorthand method of the embodiment of the present application is shown in the figure.

[0047] Figure 3 The flowchart of the conversation shorthand method of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0048] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein, on the contrary, these embodiments are provided to make the present application more thorough and complete. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes, and are not intended to limit the scope of protection of the present application.

[0049] It should be understood that each step described in the method embodiment of the present application can be executed in different order and / or in parallel. In addition, the method embodiment can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0050] The term "include," and variations thereof, as used in this document, is open-ended and means "to comprise" or "to consist of" and is not limiting; the term "based on," as used in this document, is the conventional attachment of conditions to statement of facts to allow for exceptions; the term "one embodiment" as used herein means "at least one embodiment"; the term "another embodiment" as used herein means "at least one additional embodiment"; the term "some embodiments" as used herein means "at least some embodiments"; the term "optional" as used herein means "optional but not required"; related definitions are given throughout the detailed description. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0051] It should be noted that the terms "one" and "a" as used herein are intended to denote "at least one" of the referenced elements, and are not intended to denote a single element, unless otherwise indicated.

[0052] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0053] To solve the problems in the related art, the present embodiment provides a call shorthand method.

[0054] As shown in Figure 1 and Figure 3 The call shorthand method provided by the embodiments of the present application comprises:

[0055] In step S100, when there is valid speech, voiceprint feature detection is performed on the valid speech to obtain voiceprint feature data.

[0056] When the call scene has speech signals that can be recognized and used for subsequent processing, the speech signals are taken as valid speech for voiceprint feature detection. Specifically, after starting the call, an established voiceprint model is initialized for voiceprint feature detection. When there is valid speech, voiceprint feature detection is performed on the valid speech to obtain voiceprint feature data, which is used to identify the identity of the speaker; when there is no valid speech, subsequent operations are not triggered, for example, when the speech only has noise or the energy in the speech is not enough to be determined as valid speech, no voiceprint feature detection is performed on the speech.

[0057] In step S200, identity recognition is performed on the voiceprint feature data to determine the speaker in the valid speech.

[0058] In a voice call, there can be a conversation between two or more people, and there can be noise, weak signal, or other factors that interfere with recognition. In an embodiment, the collected valid voice is pre-processed, such as removing environmental noise and silent segments, and then voiceprint feature parameters are extracted.

[0059] The voiceprint feature parameters can include but are not limited to fundamental frequency distribution, formant frequency, timbre texture, spectrogram features, etc., for characterizing the unique voice habits of the speaker. The extracted voiceprint feature parameters are converted into feature vectors as input data for identity recognition. The dimension of the feature vector can be adjusted according to actual needs, such as through Fourier transform, Mel frequency cepstral coefficient (MFCC), or deep learning model encoding generation.

[0060] The generated feature vectors are compared with reference feature vectors in a pre-set voiceprint database. The voiceprint database stores voiceprint templates of registered speakers, each template corresponding to a unique identity. The comparison process can be implemented through similarity calculation or classification model, and the matching score is output. The identity of the speaker corresponding to the current voice is determined according to the matching score. If the matching score is higher than the score threshold, it is determined to be a registered speaker in the database; if the matching score is lower than the score threshold, it is marked as an unknown speaker.

[0061] The score threshold can be dynamically adjusted according to the actual application scenario to balance the recognition accuracy and the false positive rate. The recognized speaker identity is associated with the original voice data to generate a voice segment record containing the speaker identification. This record can be used for subsequent authorization verification and shorthand processing procedures, such as distinguishing between authorized and unauthorized speaker voice content.

[0062] Step S300, generating an interaction prompt to the un-pre-authorized speaker for obtaining shorthand authorization of the speaker.

[0063] The recognized speaker identity is matched with an authorization list to determine whether the speaker is in an unauthorized state. If not authorized, the interaction prompt generation process is triggered.

[0064] Generate a standardized prompt message according to the current scenario. The prompt message can include text description, voice broadcast, or interface pop-up, etc. Output the prompt message through an interactive channel matching the current call device. The interactive channel includes but is not limited to call interface display, earphone voice playback, SMS sending, or third-party application notification. The output method needs to adapt to the device type and user preference settings.

[0065] Receive the speaker's response to the prompt message, such as through key confirmation, voice command, or interface click to select "Allow" or "Reject". Verify the validity of the response signal to exclude false touch or invalid operations.

[0066] The authorized response that passes the verification is bound with the speaker identity, and the authorized list is updated. The authorized time, authorized scope, and speaker identification are recorded to provide a basis for subsequent shorthand processing. If the speaker does not make a valid response within the specified time, it is processed according to the preset strategy. For example, the authorization is denied by default, the prompt is repeated, or the current prompt process is terminated.

[0067] In step S400, when the shorthand authorization of all speakers in the valid speech has been obtained, the speech of all speakers is transcribed.

[0068] The speech signal is converted into text content through a speech recognition model, and the timestamp information of the original speech is retained for subsequent alignment.

[0069] Based on the identity recognition result, the roles of the speech segments are divided, and the speech content of different speakers is labeled and classified. For example, through voiceprint feature matching, the continuous speech of the same speaker is merged into an independent paragraph, and the corresponding identity label (such as "Speaker A" and "Speaker B") is labeled. Syntax correction and semantic completion operations are performed on the recognized text to generate shorthand content that conforms to written expression specifications. Key information (such as time, location, and matters) is extracted through natural language processing technology, and an abstract framework is constructed according to the dialogue logic.

[0070] The shorthand text is associated with the timestamp of the original speech to form a traceable multi-modal record. For example, in the text, "[03:15] Speaker A: The meeting needs to be completed by this Friday …", it is convenient for subsequent review and verification. According to the application scenario requirements, the storage format is selected. If retrieval and analysis are required, the text content can be split into a keyword index table; if complete speech information needs to be retained, the speech file and the text transcript are saved synchronously, and they are bound by a unique identifier. Check if the shorthand content meets the privacy protection requirements (such as not containing information of unauthorized speakers).

[0071] In step S500, when the shorthand authorization of all speakers is not obtained, the audio of the authorized speakers is retained based on the voiceprint feature data, and the call transcription is performed.

[0072] Based on the identity recognition result, speakers who are not matched in the authorized list are filtered out. The speech segments of the unauthorized speakers are marked, and their sound time intervals and corresponding voiceprint feature data are recorded for filtering.

[0073] The original speech is divided into independent segments according to the speaker identity, and each segment is classified using voiceprint feature data. By comparing with the preset authorized database, only the speech content of the authorized speakers is retained, and the rest of the segments are muted or deleted.

[0074] Perform end-to-end speech recognition to generate a text transcript. Based on speaker identification, annotate the text with roles (e.g., "Speaker A:...") and extract key information to generate a summary. Save the audio file and text record synchronously, bound by timestamps.

[0075] Verify that the processed speech and text are completely free of unauthorized content and confirm that no sensitive information remains. Store authorized data separately in a designated area, physically isolated from unauthorized data, to prevent cross-access.

[0076] In this embodiment, the voiceprint feature has individual uniqueness, which can effectively distinguish different speaker identities and maintain recognition stability in different contexts. After extracting voiceprint feature data from valid speech, the data can be matched with a pre-stored voiceprint database to determine the current speaker's identity. On this basis, the identified identity information is associated with the authorization status, providing a basis for subsequent verification of whether to allow shorthand. The unpreauthorized speaker is actively sent an authorization request, which can protect the user's right to know and also meet the real-time communication and privacy protection needs. When all participants in the conversation are authorized, the entire speech content can be transcribed to generate a conversation shorthand text; if there is an unauthorized speaker, the voice segment of the unauthorized speaker can be dynamically filtered through voiceprint feature data, and only the audio and corresponding text of the authorized part are retained to avoid sensitive information leakage. Through the combination of voiceprint recognition and dynamic authorization mechanism, the strong demand of users for privacy protection is met, and the efficient recording and organization of authorized content are guaranteed.

[0077] Optionally, when the shorthand authorization of all speakers in the valid speech has been obtained, the speech of the all speakers is transcribed for the conversation shorthand, including:

[0078] The speech of the all speakers is encrypted, and a merged speech stream is obtained through a time domain interleaving algorithm.

[0079] The merged speech stream is transcribed for the conversation shorthand.

[0080] In an embodiment, a time domain interleaving algorithm is used to merge the speech streams of all speakers, and the data is encrypted through an AES-256-GCM encryption method and transmitted to the user terminal. By merging the speech streams, the speech signals of both parties are staggered in the time dimension to avoid data conflicts or overlaps, while preserving the timing information of the speech. By time staggering, transmission delay is reduced and real-time performance is improved.

[0081] Optionally, the interleaving period interval of the time domain interleaving algorithm can be set to 30-70 ms.

[0082] Optionally, after the speech of the all speakers is encrypted and a merged speech stream is obtained through a time domain interleaving algorithm, it further includes:

[0083] monitoring quality loss of the merged voice stream every preset time interval;

[0084] triggering the shorthand authorization for all speakers again when the quality loss is greater than a preset threshold.

[0085] In an embodiment, when the network quality is poor, it may affect the transmission quality of the voice stream, thereby affecting the accuracy of voiceprint recognition. Therefore, the quality loss of the merged voice stream is monitored every preset time interval. If the current quality loss affects the accuracy of voiceprint recognition, the preset threshold is used to determine the possibility. If the quality loss is greater than the preset threshold, the shorthand authorization request for all speakers is triggered again to adjust the transmission strategy.

[0086] Adjusting the transmission strategy includes:

[0087] canceling the encryption transmission process, i.e., canceling the call shorthand of the call, ensuring voice call chat, and not performing subsequent voice separation and transcription processes;

[0088] bandwidth and code rate adaptive adjustment, i.e., dynamically reducing the voice coding code rate (such as from 12.2 kbps to 6.4 kbps) according to the network condition, sacrificing part of the sound quality to improve transmission stability;

[0089] transmission channel switching, i.e., automatically switching to a backup communication link such as switching from 4G to Wi-Fi or enabling Mesh relay transmission.

[0090] Optionally, the preset time interval can be set according to requirements. For example, the preset time interval is set according to the network condition. When the network condition is poor, the preset time interval is set to 5-10 s. When the network condition is good, the preset time interval is set to 30-60 s.

[0091] Optionally, the monitoring of the quality loss of the merged voice stream every preset time interval includes:

[0092] monitoring the word error rate and the character error rate of the merged voice stream.

[0093] In an embodiment, the WER (word error rate) and CER (character error rate) of the merged voice stream are monitored to obtain the quality loss of the merged voice during transmission. When the word error rate is greater than a preset threshold or the character error rate is greater than a preset threshold, it indicates that the transmission quality may affect the accuracy of subsequent voiceprint recognition, and re-authorization is required.

[0094] When the word error rate and the character error rate are both less than a preset threshold, the merged voice stream is normally stored and processed.

[0095] Optionally, before performing voiceprint feature detection on the valid speech when valid speech exists to obtain voiceprint feature data, the method further includes:

[0096] Acquire the voice recording of the call, and take the voice stream that meets the valid voice conditions as the valid voice, wherein the valid voice conditions include at least one of the following: the duration of the voice exceeds a preset duration, the voice frequency meets a preset fundamental frequency range, and the voice harmonic structure meets a preset harmonic structure.

[0097] In one embodiment, the existence of valid speech is determined by valid speech conditions, wherein the duration of the sound exceeds a preset duration to exclude instantaneous noise interference; the sound frequency needs to conform to the fundamental frequency range of human voice, such as 80-400Hz; and the sound harmonic structure needs to meet the harmonic structure of human voice to avoid misjudging non-speech signals.

[0098] Optionally, the preset duration can be set to 100-300ms. For example, if it is set to 200ms, the sound duration exceeding 200ms is considered non-transient noise.

[0099] Optionally, valid speech conditions also include a normalized energy value exceeding the background noise.

[0100] In one embodiment, if the normalized energy value exceeds the background noise by more than 3dB, the speech frame is determined to have valid speech.

[0101] Optionally, such as Figure 2 As shown, before performing voiceprint feature detection on the valid speech when valid speech exists to obtain voiceprint feature data, the method further includes:

[0102] Step S010: Perform echo cancellation and noise suppression on the effective speech, and segment it according to the preset window length to obtain speech segments.

[0103] Step S020: Perform a first-order high-pass filter on the speech segment to obtain pre-emphasized speech.

[0104] Step S030: The number of pre-emphasized speech segments is taken as the number of samples, the number of vocal tracts of the pre-emphasized speech segments is taken as the number of channels, and the frequency dimension and time dimension of the pre-emphasized speech segments are taken as the height and width, respectively, and fed into the convolutional neural network to obtain the silence detection threshold.

[0105] Step S040: Based on the silence detection threshold, perform silence detection on the pre-emphasized speech, remove silent segments, and obtain silence detection results.

[0106] Step S050: Perform zero-crossing rate detection on the pre-emphasized speech to obtain the zero-crossing rate detection result.

[0107] Step S060, obtaining a voice activity detection decision result according to the mute detection result and the zero-crossing rate detection result.

[0108] In an embodiment, before performing the voiceprint feature detection, the valid speech needs to be preprocessed to better extract the features in the valid speech. The preprocessing process includes echo cancellation, noise suppression, and voice activity detection decision based on a convolutional neural network model.

[0109] Specifically, the valid speech is obtained, and an echo cancellation algorithm is performed on the valid speech to suppress the echo of the loudspeaker, and an ambient noise reduction algorithm is used to reduce the ambient noise. After noise suppression, the valid speech is divided into speech segments with a fixed window length and a preset frame shift according to a preset window length.

[0110] The processing formula can be expressed as:

[0111] w(n) = 0.54 - 0.46cos(2πn / N),

[0112] where w(n) represents the weight value of the window function at the nth sampling point, n represents the position of the current sampling point, and the value range is 0≤n<N, i.e. the length of the window, and N represents the length of the window.

[0113] Optionally, the length of the window is set to 20 ms.

[0114] The valid speech is divided into short-time frames with a fixed window length of 20 ms and a frame shift of 10 ms, so that each frame overlaps by 10 ms to ensure smooth transition between frames. The short-time speech segments are pre-emphasized by a Hamming window to suppress spectral leakage, and pre-emphasized by a first-order high-pass filter, which is expressed as:

[0115] y(n) = x(n) - 0.97x(n-1),

[0116] x(n) represents the value of the input original speech signal at the nth sampling point, y(n) represents the value of the output signal after pre-emphasis processing at the nth sampling point, 0.97 represents the pre-emphasis coefficient, and the value range is usually 0.9<α<1.0, 0.97 represents the commonly used default value, and x(n-1) represents the sampling value at the previous time, which is used to calculate the difference between the current time and the previous time.

[0117] In an embodiment, the pre-emphasized speech is processed by a convolutional neural network to automatically extract the distinguishing features of speech / noise. The convolutional neural network needs four parameters B, C, H, and W as input, and the number of audio samples processed in batches is taken as the input B of the convolutional neural network,

[0118] The number of channels corresponding to the audio is taken as the input C of the convolutional neural network, wherein the single-channel value is 1 and the stereo value is 2, the frequency dimension, i.e., the vertical resolution of the spectrogram, is taken as the input H of the convolutional neural network, and the time dimension, i.e., the horizontal resolution of the spectrogram, is taken as the input W of the convolutional neural network.

[0119] The convolutional neural network outputs a corresponding silence detection threshold Eth according to the input pre-emphasis voice, and the short-time energy of the pre-emphasis voice is compared with the silence detection threshold Eth to perform silence detection, and the silence segment is removed. The short-time energy of the pre-emphasis voice is represented as:

[0120] ,

[0121] wherein E n represents the short-time energy, N represents the frame length, i.e., the number of samples in the window, k represents the sample point sequence number, and x k represents the signal amplitude at the kth time point.

[0122] All samples x k are squared to eliminate the positive and negative effects, and all squared values are accumulated to obtain the short-time energy E n . E n is compared with the silence detection threshold Eth to determine whether it is silence. When E n is less than the silence detection threshold, it is determined that it is silence at this time, and when E n is greater than or equal to the silence detection threshold, it is determined that the pre-emphasis voice has sound.

[0123] Optionally, when the silence detection threshold output by the convolutional neural network is abnormal, the silence detection threshold is set to a preset silence detection threshold, wherein the value of the preset silence detection threshold is 0.3.

[0124] In an embodiment, when the silence detection threshold output by the convolutional neural network is too large or too small, it is determined that the convolutional neural network is abnormal, the silence detection threshold output does not have reference value, and then the silence detection is performed through the preset silence detection threshold.

[0125] At the same time of performing the silence detection, the zero-crossing rate detection can be performed on the pre-emphasis voice to calculate the number of times of sign changes of adjacent sample points, which is used to assist in constructing the voice activity detection decision.

[0126] The zero-crossing rate detection is represented as:

[0127] ,

[0128] ,

[0129] wherein, ZCR represents a zero-crossing rate, N represents a frame length, i.e., a number of samples in a window, k represents a sample point serial number, sgn(*) represents a sign function of x k The equation group is constructed.

[0130] In the embodiment, whether the current speech has human voice is determined by the zero-crossing rate threshold value, when the zero-crossing rate is greater than or equal to the zero-crossing rate threshold value, it is determined that the pre-emphasis speech does not have human voice, and when the zero-crossing rate is less than the zero-crossing rate threshold value, it is determined that the pre-emphasis speech has human voice.

[0131] Logical operation is performed on the silence detection result and the zero-crossing rate detection result, when the silence detection result is non-silence and the zero-crossing rate detection result shows that there is human voice, it is determined that the current frame is speech activity, and a speech activity detection decision result is output as speech;

[0132] When the silence detection result is non-speech or the zero-crossing rate detection result is non-speech, it is determined that the current frame is non-speech activity, and a speech activity detection decision result is output as non-speech.

[0133] Optionally, the convolutional neural network also outputs a zero-crossing rate threshold value.

[0134] In an embodiment, after the pre-emphasis speech is processed by the convolutional neural network, a silence detection threshold value and a zero-crossing rate threshold value are output, whether the pre-emphasis speech includes human voice is determined by the silence detection threshold value and the zero-crossing rate threshold value, and a speech activity detection decision result is generated.

[0135] The speech activity detection decision result includes: weighting the silence detection threshold value and the zero-crossing rate threshold value respectively, outputting a probability value of the existence of speech (such as P(speech)=0.8), and determining as speech or non-speech by setting a threshold value; after the silence detection threshold value and the zero-crossing rate threshold value are weighted and added, binary classification is performed, and a binary label (0 / 1) is output, indicating non-speech / speech.

[0136] In a real-time call scenario, the speech activity detection decision is used for dynamically controlling a recording range (such as recording only user speech and filtering opponent speech).

[0137] Optionally, when there is valid speech, voiceprint feature detection is performed on the valid speech to obtain voiceprint feature data, including:

[0138] A multi-channel triangular filter is constructed.

[0139] A feature vector of the valid speech is extracted according to the multi-channel triangular filter, wherein the feature vector includes a mel-frequency cepstral coefficient, a first-order differential mel-frequency cepstral coefficient, and an energy feature.

[0140] The feature vector is dimensionally optimized based on a supervised dimension reduction algorithm to obtain the voiceprint feature data.

[0141] Optionally, the multi-channel triangular filter is a 26-channel triangular filter, used to extract 12-dimensional MFCC (Mel Frequency Cepstrum Coefficient), 12-dimensional delta MFCC (first-order differential Mel frequency cepstrum coefficient) + 2-dimensional energy features, represented as:

[0142] f m =2595log 10 (1+m / 700),

[0143] wherein f m represents the feature dimension, and m takes a value of 1-26.

[0144] Based on linear discriminant analysis (LDA), the 26-dimensional MFCC feature is optimized to 40 dimensions, and the feature dimension with the strongest speaker distinguishability is retained.

[0145] Optionally, the identity recognition of the voiceprint feature data to determine the speaker in the valid voice includes:

[0146] Based on the voiceprint change point, the voiceprint segment is divided, wherein the voiceprint change point includes a mute point and a sound energy mutation point.

[0147] Through a clustering algorithm, similar voice features are classified as the same speaker in the voiceprint feature data corresponding to the voiceprint segment.

[0148] The energy of the voiceprint segment is calculated, and an energy threshold is set. When the energy of the voiceprint segment is less than the mute detection threshold, the voiceprint segment is determined as a mute segment. The continuous frame sequence corresponding to the mute point constitutes a mute segment.

[0149] The energy difference of adjacent voiceprint segments is calculated. When the energy difference exceeds the mutation threshold, the voiceprint segment is determined as a sound energy mutation point. The mutation point can represent the sudden enhancement or weakening of the voice signal, and is used to divide the boundary of different speakers or sentences.

[0150] Optionally, the clustering algorithm includes K-means, GMM algorithm.

[0151] Optionally, the interaction prompt includes at least one of a visual interaction interface or a text prompt.

[0152] After the interaction prompt is generated to the un-pre-authorized speaker, it further includes:

[0153] In response to a shorthand authorization instruction, an encrypted channel is established, and the shorthand authorization state of the speaker is changed according to the shorthand authorization instruction.

[0154] In an embodiment, the visual interactive interface includes a semi-transparent floating box, and the semi-transparent floating box is arranged at x=screen width-200px and y=screen height-100px in the lower right corner of the video picture.

[0155] The text prompt includes a dynamic prompt script, and the initial state is: press # key to authorize the other party's voice recording (RGB color value #FF6B6B, transparency 80%), and the authorized state is: authorized (green check mark, animation duration 300ms).

[0156] The key detection includes physical key detection and virtual key detection, wherein the physical key detection includes physical keys provided by the terminal, such as detection by triggering the # key; the virtual key detection includes detection by a virtual button on the interactive unit of the terminal, for example, when the interactive unit is a touch screen, the shorthand authorization instruction is generated by a touch signal at a preset position of the touch screen.

[0157] After the key is triggered, a DTLS (Datagram Transport Layer Security) encrypted channel is established, and the authorization instruction is transmitted to the terminal.

[0158] Optionally, when the shorthand authorization of all speakers is not obtained, the audio of the authorized speaker is retained according to the voiceprint feature data, and the call shorthand is performed, including:

[0159] Obtain the authorized voiceprint feature data of the authorized speaker;

[0160] The audio corresponding to the authorized voiceprint feature data greater than the preset likelihood ratio is taken as the target audio;

[0161] The audio corresponding to the authorized speaker is subjected to the call shorthand.

[0162] In an embodiment, when the shorthand authorization of all speakers is not obtained, only the authorized voice stream is subjected to the call shorthand, and by calculating the likelihood ratio between the authorized voiceprint feature data and the audio, the audio with a likelihood ratio greater than a preset likelihood ratio is taken as the target audio and subjected to the call shorthand.

[0163] The likelihood ratio formula is represented as:

[0164] ,

[0165] wherein, represents the likelihood ratio, s e represents the authorized speaker model, s u represents the general background model, p(x|s e ) represents that the input voice feature belongs to the authorized speaker (se ) represents the probability of the input speech feature belonging to the general background model (s u ) represents the probability of the input speech feature belonging to the general background model (s u ) represents the probability of the input speech feature belonging to the general background model (s

[0166] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose authorization or refusal.

[0167] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A method for shorthand recording of phone calls, characterized in that, include: When valid speech is available, voiceprint feature detection is performed on the valid speech to obtain voiceprint feature data; The voiceprint feature data is used for identity recognition to determine the speaker in the valid speech; Generate interactive prompts to the speaker who has not pre-authorized the service, in order to obtain the speaker's shorthand authorization; When the shorthand authorization for all speakers in the valid voice has been obtained, call shorthand is performed on the voice of all speakers, including: encrypting the voice of all speakers and obtaining a merged voice stream through a time-domain interleaving algorithm; monitoring the quality loss of the merged voice stream at preset time steps; when the quality loss is greater than a preset threshold, the shorthand authorization for all speakers is triggered again; and call shorthand is performed on the merged voice stream. When the shorthand authorization of all speakers is not obtained, the audio of the authorized speakers is retained based on the voiceprint feature data, and the call shorthand is performed.

2. The call shorthand method according to claim 1, characterized in that, The step of monitoring the quality loss of the merged voice stream at preset time steps includes: The word error rate and character error rate of the merged speech stream are monitored.

3. The call shorthand method according to claim 1, characterized in that, Before performing voiceprint feature detection on the valid speech when valid speech is present to obtain voiceprint feature data, the method further includes: Acquire the voice recording of the call, and take the voice stream that meets the valid voice conditions as the valid voice, wherein the valid voice conditions include at least one of the following: the duration of the voice exceeds a preset duration, the voice frequency meets a preset fundamental frequency range, and the voice harmonic structure meets a preset harmonic structure.

4. The call shorthand method according to claim 1, characterized in that, Before performing voiceprint feature detection on the valid speech when valid speech is present to obtain voiceprint feature data, the method further includes: The valid speech is subjected to echo cancellation and noise suppression, and segmented according to a preset window length to obtain speech segments; The speech segment is subjected to a first-order high-pass filter to obtain pre-emphasized speech; The number of pre-emphasized speech segments is used as the number of samples, the number of vocal tracts of the pre-emphasized speech segments is used as the number of channels, and the frequency dimension and time dimension of the pre-emphasized speech segments are used as the height and width, respectively. These are then fed into a convolutional neural network to obtain the silence detection threshold. Based on the silence detection threshold, silence detection is performed on the pre-emphasized speech, silence segments are removed, and silence detection results are obtained. Zero-crossing rate detection is performed on the pre-emphasized speech to obtain the zero-crossing rate detection result; The speech activity detection decision result is obtained based on the silence detection result and the zero-crossing rate detection result.

5. The call shorthand method according to claim 1, characterized in that, When valid speech is available, performing voiceprint feature detection on the valid speech to obtain voiceprint feature data includes: Construct a multi-channel triangular filter; The feature vector of the effective speech is extracted according to the multi-channel triangular filter, wherein the feature vector includes Mel frequency cepstral coefficients, first-order difference Mel frequency cepstral coefficients, and energy features; The feature vector is optimized for dimension using a supervised dimensionality reduction algorithm to obtain the voiceprint feature data.

6. The call shorthand method according to claim 1, characterized in that, The step of performing identity recognition on the voiceprint feature data to determine the speaker in the valid speech includes: Voiceprint segments are divided based on voiceprint change points, wherein the voiceprint change points include silence points and sound energy change points; Using a clustering algorithm, voices with similar voiceprint features are classified as belonging to the same speaker in the voiceprint feature data corresponding to the voiceprint segments.

7. The call shorthand method according to claim 1, characterized in that, The interactive prompts include at least one of a visual interactive interface or a text prompt; After generating interactive prompts for the unauthorized speaker, the method further includes: In response to a shorthand authorization instruction, an encrypted channel is established, and the speaker's shorthand authorization status is changed according to the shorthand authorization instruction.

8. The call shorthand method according to claim 1, characterized in that, When not obtaining shorthand authorization from all speakers, the step of preserving the audio of the authorized speakers based on the voiceprint feature data and performing call shorthand includes: Obtain the authorized voiceprint feature data of the authorized speaker; The audio corresponding to the authorized voiceprint feature data that has a greater than preset likelihood ratio is taken as the target audio; The audio corresponding to the authorized speaker is used for call stenography.

Images