A recording automatic generation method based on AI voice interaction

Abstract

The application discloses a kind of based on AI voice interaction's record automatic generation method, belong to voice analysis technical field, comprising: obtaining original voice data, continuous audio stream is recorded from target environment by preset audio acquisition module, generates the voice segment set of preliminary division using time domain segmentation technique;For the voice segment set, extract pitch feature, speech rate variation and volume feature, generate voice feature set;The voice feature set is input into deep neural network model, determine emotional label by multidimensional feature mapping, generate voice unit set with emotional label;If the voice type of voice unit set is statement type, then use narrative template to generate text;If the voice type of voice unit set is inquiry type, then use inquiry template to generate text;Get formatted record.The based on AI voice interaction's record automatic generation method solves the problem that existing record generation mode is difficult to generate format standard and information-rich record.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech analysis, in particular to a recording automatic generation method based on AI voice interaction. BACKGROUND

[0002] The generation of interrogation records is a crucial step in the judicial field, directly related to the efficiency of case investigation and the credibility of evidence. However, current reliance on manual recording or other semi-automatic solutions has significant limitations. Manual recording is time-consuming, inefficient, and prone to missing critical information or misjudging speech types due to the subjective judgment of the recorder.

[0003] In the field of interrogation, the main challenges focus on the accuracy of speech type recognition and the intelligent capture of emotional information. In the interrogation scene, speech is often accompanied by background noise, and the emotional information contained in the tone, speed, and volume of speech changes frequently. Some existing automated tools can achieve basic speech-to-text functions, but often lack the ability to adapt to complex speech scenarios, especially in the aspect of emotional tagging, which makes it difficult to generate records with standardized format and rich information, limiting its application value in actual interrogation. SUMMARY

[0004] In order to overcome the defects existing in the prior art, the present application provides a recording automatic generation method based on AI voice interaction to solve the above problems.

[0005] The technical solution adopted by the present application to solve its technical problems is: a recording automatic generation method based on AI voice interaction, comprising the following steps:

[0006] S1: Obtain the original speech data, record continuous audio stream from the target environment through the preset audio acquisition module, and generate a set of initially divided speech segments using time domain segmentation technology;

[0007] S2: For the set of speech segments, extract tone features, speed changes, and volume features to generate a set of speech features;

[0008] S3: Input the set of speech features into a deep neural network model to determine the emotional label through multi-dimensional feature mapping, and generate a set of speech units with emotional labels;

[0009] S4: If the speech type of the speech unit set is statement type, use narrative templates to generate text; if the speech type of the speech unit set is inquiry type, use inquiry templates to generate text; obtain the formatted record.

[0010] Preferably, in the step S1, the original speech data is time domain separated according to the preset silence threshold to generate a set of speech segments with timestamps.

[0011] Optionally, in the step S1, the duration of each segment in the set of speech segments is obtained; if the duration exceeds a preset threshold, a short-time energy-based endpoint detection algorithm is used to reposition the segmentation points of the segment to optimize the set of speech segments.

[0012] Specifically, in the step S1, the mel-frequency cepstral coefficients of each segment are extracted from the optimized set of speech segments;

[0013] The mel-frequency cepstral coefficients are classified using a pre-trained classifier to obtain the class label of each segment;

[0014] Segments with speaking class labels are extracted, and consecutive segments with speaking class labels are merged to further optimize the set of speech segments.

[0015] Specifically, in the step S2, short-time Fourier transform processing is performed on the speech segments in the set of speech segments to obtain a time-frequency matrix corresponding to each frame in the speech segment, the fundamental frequency trajectory is extracted from the time-frequency matrix as the pitch feature, the energy mean of each frame is calculated as the volume feature, and the speech rate change feature is obtained through syllable boundary detection.

[0016] It is worth noting that in the step S2, the pitch feature, volume feature and speech rate change feature are combined into a three-dimensional vector, which is input into a Gaussian mixture model to obtain a clustering label. The pitch feature, volume feature and speech rate change feature are bound to the corresponding clustering label to form a speech feature set.

[0017] Optionally, in the step S3, the speech feature set is analyzed by a deep neural network to obtain a speech feature vector;

[0018] According to the speech feature vector, a principal component analysis technique is used to extract a change type to obtain preliminary emotional change data;

[0019] The preliminary emotional change data is input into a pre-trained convolutional neural network to determine the emotion label of each frame in the speech segment, and then the speech segments with emotion labels are recombined into a speech unit set.

[0020] Specifically, in the step S4, the speech segments in the speech unit set are obtained, and the pitch feature, volume feature and speech rate change feature of the speech segments are combined to determine whether the speech segments belong to the statement type or the inquiry type to obtain a type judgment result.

[0021] If the type judgment result is a statement type, the voice segment is converted into text and filled into a narrative template, and the corresponding emotional label is attached; if the type judgment result is an inquiry type, the voice segment is converted into text and filled into an inquiry template, and the corresponding emotional label is attached; and a formatted record is obtained.

[0022] The present application has the beneficial effect that in the AI voice interaction-based record automatic generation method, the recorded continuous audio stream is preliminarily divided into a voice segment set by a time domain segmentation technique, and then the voice feature set is extracted in the voice segment set, and then the voice feature set is input into a deep neural network model to determine the emotional label, and a voice unit set with an emotional label is generated. Based on this voice unit set, the corresponding template is selected to generate a formatted record according to different voice types. This scheme realizes the automatic processing from the original voice to the formatted record, and generates a record with standardized format and rich information, which guarantees its application value in actual interrogation. BRIEF DESCRIPTION OF DRAWINGS

[0023] Fig. 1 The flowchart of the AI voice interaction-based record automatic generation method in an embodiment of the present application;

[0024] Fig. 2 The generation flowchart of the voice feature set in an embodiment of the present application;

[0025] Fig. 3 The generation flowchart of the voice unit set in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present application, but does not constitute a limitation on the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0027] As shown in Figs. 1-3 , an AI voice interaction-based record automatic generation method comprises the following steps:

[0028] S1: obtaining original voice data, recording a continuous audio stream from a target environment through a preset audio acquisition module, and generating a preliminarily divided voice segment set by a time domain segmentation technique;

[0029] S2: extracting pitch features, speech rate changes and volume features for the voice segment set, and generating a voice feature set;

[0030] S3: inputting the voice feature set into a deep neural network model, determining an emotion label through multi-dimensional feature mapping, and generating a voice unit set with the emotion label;

[0031] S4: if the voice type of the voice unit set is a statement type, generating text by using a narrative template; if the voice type of the voice unit set is an inquiry type, generating text by using an inquiry template; and obtaining a formatted record.

[0032] In the AI voice interaction-based record automatic generation method, the recorded continuous audio stream is preliminarily divided into a voice segment set through a time domain segmentation technique, and then a voice feature set is extracted from the voice segment set. Subsequently, the voice feature set is input into a deep neural network model to determine an emotion label and generate a voice unit set with the emotion label. Based on the voice unit set, a corresponding template is selected to generate a formatted record according to different voice types. The present scheme realizes automatic processing from the original voice to the formatted record, and generates a record that is formatted and information-rich, thereby ensuring its application value in actual interrogation.

[0033] It is worth noting that in the step S1, the original voice data is time-domain segmented according to a preset silence threshold to generate a voice segment set with timestamps.

[0034] In the present embodiment, the target environment audio stream is collected by a microphone array, and original audio data with a preset sampling rate is obtained. For example, in an interrogation room, the microphone array can be deployed in the center of the room to collect surrounding voice signals. Assuming that the sampling rate is preset to 16 kHz to ensure that the audio quality is suitable for subsequent processing. Such a sampling rate is relatively common in voice processing and can balance the clarity and data volume. The original audio data is time-domain segmented by a silence detection module, and a voice segment set with timestamps is generated according to a preset silence threshold. For example, the silence detection module can be set to a silence threshold of -40 dB, and audio below this value is considered to be a silence area. When someone speaks in the interrogation, the voice part will be retained, and below the silence threshold, it is considered to be a gap without speaking, which will be marked as silence, and multiple segments with timestamps are generated based on this. The advantage of this is to preliminarily separate the effective voice and reduce the subsequent computational burden.

[0035] Preferably, in the step S1, the duration of each segment in the voice segment set is obtained; if the duration exceeds a preset threshold, a short-time energy-based endpoint detection algorithm is used to reposition the segmentation point of the segment to optimize the voice segment set.

[0036] For a set of speech segments, the duration of each segment is calculated. In one possible implementation, it is assumed that the duration of a segment is 4 seconds, and the preset threshold is 3 seconds. The segments exceeding the preset threshold need to be further processed. For example, if a person speaks for a long time in an interrogation, the segment may contain complete sentences or even multiple sentences, and the duration is relatively long. Therefore, more fine segmentation is needed to improve the subsequent classification accuracy. If the duration exceeds the preset threshold, an endpoint detection algorithm based on short-time energy is used to reposition the segmentation points to obtain an optimized set of speech segments. The endpoint detection algorithm based on short-time energy can detect the start and end points of the speech segment. For example, in a 4-second segment, the actual speech in the segment starts at 0.5 seconds and ends at 3.5 seconds by judging the energy peak value. The first and last silences are removed, and the optimized segment is more consistent with the true speech range. This method can effectively improve the purity of the speech segment and lay a foundation for feature extraction.

[0037] Optionally, in the step S1, the mel-frequency cepstral coefficients of each segment are extracted from the optimized set of speech segments. It can be understood that the mel-frequency cepstral coefficients simulate the perception of the human ear to the frequency and are commonly used in speech recognition.

[0038] The mel-frequency cepstral coefficients are classified by using a pre-trained classifier to obtain the class label of each segment. The mel-frequency cepstral coefficients are classified by using a pre-trained classifier to obtain the class label of each segment. In one embodiment, the classifier can be a support vector machine, which can distinguish labels such as “speech”, “noise” and “background sound” after pre-training. For example, a certain segment of audio in an interrogation is labeled as “speech”, and another segment is labeled as “noise” because it contains fan sound. This classification helps to understand the audio content and improve the semantic value of the processing result.

[0039] The segments with the class label of speech are extracted, and the continuous segments with the class label of speech are merged to further optimize the set of speech segments. For example, a person speaks three sentences continuously in an interrogation, and the three sentences are labeled as “speech” in the corresponding single segment and three continuous segments. Therefore, the three segments can be merged into a complete speech segment and labeled as “speech”. This merging can reduce fragmentation, improve the coherence and practicality of the result, and facilitate archiving or subsequent speech-to-text. In one possible implementation, the entire process from collection to classification and then to merging forms an efficient audio processing chain. For example, the audio in a noisy environment in an interrogation room can be clearly separated into speech content and noise after the above processing.

[0040] Specifically, in the step S2, the short-time Fourier transform processing is performed on the speech segments in the set of speech segments to obtain a time-frequency matrix corresponding to each frame in the speech segment. The fundamental frequency trajectory is extracted from the time-frequency matrix as a pitch feature, the energy mean value of each frame is calculated as a volume feature, and the syllable boundary detection is used to obtain the speech rate change feature.

[0041] In the embodiment, the Hann window is used for short-time Fourier transform processing of the voice segment. It can be understood that the Hann window is a smooth window function, which can effectively reduce spectral leakage and improve the accuracy of time-frequency analysis. For example, in an interrogation recording, a 2-second voice segment is divided into multiple 20-millisecond frames, each frame is applied with a Hann window and then subjected to short-time Fourier transform to generate a time-frequency matrix.

[0042] Specifically, the time-frequency matrix reflects the energy distribution of the voice segment in time and frequency, such as the energy of a certain frame at 300 Hz is strong, which is considered as the fundamental frequency of the speaker. The fundamental frequency trajectory is extracted from the time-frequency matrix as the pitch feature, so as to track the main frequency change of each frame; for example, the pitch of a person in the interrogation rises from 250 Hz to 350 Hz, and the fundamental frequency trajectory can reflect this fluctuation. The energy mean value of each frame is calculated as the volume feature, which is more intuitive. In a possible implementation manner, the energy mean value of a certain frame of voice may be -20 dB, indicating that the volume is moderate. When the speech rate change feature is obtained through syllable boundary detection, the recognition of the syllable boundary depends on the intensity change in the voice; specifically, a preset intensity threshold is used to determine the syllable boundary position of the voice segment, the time interval is extracted from the syllable boundary position, and the time difference of the time interval between adjacent boundaries is calculated to obtain the rhythm difference. For example, a person in the interrogation speaks 4 syllables per second, and another segment decreases to 2 syllables due to more pauses, and the speech rate change feature can capture this rhythm difference.

[0043] It is worth noting that in the step S2, the pitch feature, the volume feature and the speech rate change feature are combined into a three-dimensional vector, which is input into a Gaussian mixture model to obtain a clustering label, and the pitch feature, the volume feature and the speech rate change feature are bound to the corresponding clustering label to form a voice feature set. After the pitch feature, the volume feature and the speech rate change feature are combined into a three-dimensional vector, they are input into a Gaussian mixture model for clustering. For example, the three-dimensional vector of a certain voice segment may be [300 Hz, -15 dB, 3 syllables / second], which is input into a Gaussian mixture model for clustering to obtain 300 Hz representing "stable emotion speaking", -15 dB representing "moderate volume speaking", and 3 syllables / second representing "more pauses".

[0044] Preferably, in the step S3, the voice feature set is analyzed by a deep neural network to obtain a voice feature vector; in a possible implementation manner, when the voice feature corresponding to each voice segment in the voice feature set is analyzed by a deep neural network, a recurrent neural network can be used to process the time sequence information to generate a voice feature vector; for example, the voice feature corresponding to a 2-second voice segment is input into a deep neural network, and an 128-dimensional vector is output, representing the pitch, volume and speech rate, etc. Specifically, the deep neural network may focus on the change trend of the pitch from low to high in a certain voice segment, and the value of a certain dimension in the generated vector rises from 0.3 to 0.7, reflecting the dynamic characteristics.

[0045] According to the voice feature vector, a principal component analysis technique is used to extract a change type to obtain preliminary emotional change data; when the principal component analysis technique is used to extract the change type, the principal component analysis can reduce the high-dimensional features to the main change direction; for example, the first three principal components are extracted from the 128-dimensional vector, which respectively represent the pitch fluctuation, the speed of speech, and the volume strength, to obtain the preliminary emotional change data.

[0046] The preliminary emotional change data is input into a pre-trained convolutional neural network to determine the emotional label of each frame in the voice segment, and then the voice segment with the emotional label is recombined, and all voice segments are combined into a voice unit set. When the preliminary emotional change data is input into the convolutional neural network, the network captures local features through the convolutional layer to determine the emotional label; for example, a frame of a voice segment is labeled as "excited emotion" because the convolution kernel recognizes the pattern of pitch rising and speed increasing; in an embodiment, a 2-second voice segment is divided into 5 frames, which are respectively labeled as "calm", "excited", "excited", "calm", and "excited", and the 5 frames with emotional labels are recombined into a voice segment, thereby generating a voice segment with emotional labels.

[0047] Optionally, in the step S4, the voice segment in the voice unit set is obtained, and the pitch feature, the volume feature, and the speed change feature of the voice segment are combined to determine whether the voice segment belongs to a statement type or an inquiry type, to obtain a type determination result.

[0048] If the type determination result is the statement type, the voice segment is converted into text and filled into a narrative template, and the corresponding emotional label is attached; if the type determination result is the inquiry type, the voice segment is converted into text and filled into an inquiry template, and the corresponding emotional label is attached; and a formatted record is obtained.

[0049] It can be understood that the statement type is usually mainly for information transmission, the tone is smooth and the logic is coherent, while the inquiry type has an inquiry intention, the speed may be slightly faster and often accompanied by interrogative words.

[0050] For example, in a certain interrogation recording, a person says that the speed is stable at 3 syllables per second and the tone does not fluctuate obviously when self-describing, which can be preliminarily judged as the statement type. On the contrary, if a person says that the speed reaches 4 syllables per second and the tone rises at the end when questioning, it tends to be the inquiry type. Specifically, key words such as "what" and "how" often point to inquiry, while "already" and "therefore" often imply statement. It is worth noting that converting a voice segment into text information is a conventional technology, and will not be described again. In a possible implementation manner, the narrative template is usually designed as a structure of "answer:", and the inquiry template is usually designed as a structure of "ask:".

[0051] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, replacements, and variations of the embodiments can be made without departing from the principles and spirit of the present application, and such changes, modifications, replacements, and variations fall within the scope of the present application.

Claims

1. An AI voice interaction-based record automatic generation method, characterized in that, The method comprises the following steps: S1: obtaining original voice data, recording continuous audio stream from target environment through preset audio acquisition module, and generating preliminary divided voice segment set by using time domain segmentation technology; In the step S1, the original voice data is time domain separated according to preset mute threshold, and a voice segment set with timestamp is generated; The duration of each segment in the voice segment set is obtained; If the duration exceeds the preset threshold, a short-time energy-based endpoint detection algorithm is used to reposition the segmentation point of the segment, so as to optimize the voice segment set; Mel-frequency cepstral coefficients of each segment are extracted from the optimized voice segment set; The Mel-frequency cepstral coefficients are classified by using a pre-trained classifier, and a category label of each segment is obtained; Segments with the category label of speaking are extracted, and continuous segments with the category label of speaking are combined, so as to further optimize the voice segment set; S2: extracting pitch feature, speech rate change and volume feature from the voice segment set, and generating a voice feature set; S3: inputting the voice feature set into a deep neural network model, determining an emotion label through multi-dimensional feature mapping, and generating a voice unit set with the emotion label; In the step S3, the voice feature set is analyzed by the deep neural network, and a voice feature vector is obtained; According to the voice feature vector, a change type is extracted by using principal component analysis technology, and preliminary emotion change data is obtained; The preliminary emotion change data is input into a pre-trained convolutional neural network, the emotion label of each frame in the voice segment is determined, the voice segment with the emotion label is recombined, and all voice segments are combined into a voice unit set; S4: if the voice type of the voice unit set is a statement type, a narrative template is used to generate a text; if the voice type of the voice unit set is an inquiry type, an inquiry template is used to generate a text; A formatted record is obtained.

2. The AI voice interaction-based recording automatic generation method according to claim 1, characterized in that: In the step S2, the voice segments in the voice segment set are processed by short-time Fourier transform, a time-frequency matrix corresponding to each frame in the voice segment is obtained, a fundamental frequency trajectory is extracted from the time-frequency matrix as the pitch feature, an energy mean value of each frame is calculated as the volume feature, and a syllable boundary detection is used to obtain the speech rate change feature.

3. The AI voice interaction-based recording automatic generation method according to claim 2, characterized in that: In the step S2, the pitch feature, the volume feature and the speech rate change feature are combined into a three-dimensional vector, input into a Gaussian mixture model, and a clustering label is obtained. The pitch feature, the volume feature and the speech rate change feature are bound to the corresponding clustering label to form a voice feature set.

4. The AI voice interaction-based recording automatic generation method according to claim 1, characterized in that: In the step S4, the voice segments in the voice unit set are obtained, the pitch feature, the volume feature and the speech rate change feature of the voice segments are combined to determine whether the voice segments belong to the statement type or the inquiry type, and a type judgment result is obtained; If the type judgment result is the statement type, the voice segments are converted into a text and filled into the narrative template, and the corresponding emotion label is added; if the type judgment result is the inquiry type, the voice segments are converted into a text and filled into the inquiry template, and the corresponding emotion label is added; and a formatted record is obtained.

Images