A recording channel segmentation method, device and storage medium

Abstract

The present application provides a method, device and storage medium for recording channel segmentation. It includes: obtaining a recording to be segmented of a monophonic recording, wherein the recording to be segmented includes recording segments of at least two recording objects. Extracting spectral features from the recording to be segmented, and inputting the spectral features into a trained recording segmentation model, and outputting arrangement information of at least two recording objects and an embedding vector array through the recording segmentation model. According to the embedding vector array and the arrangement information in each time block, a non-silent time block is obtained, and a distance matrix is ​​generated according to the embedding vector array corresponding to the non-silent time block. According to the distance matrix and the prior information related to the number of recording objects, the labeling information of each time block is determined by a hierarchical clustering algorithm. According to the labeling information, an RTTM file corresponding to the recording to be segmented is generated. According to the RTTM file, a recording after channel segmentation is generated. The method of the present application optimizes the storage and processing of telephone recordings.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a method, device and storage medium for recording channel segmentation. Background Art

[0002] In the information age, phone recordings have become a critical information resource, with their application value becoming increasingly prominent across multiple fields. In diverse scenarios such as customer service and consulting services, call recordings are widely retained and used as audio data for subsequent in-depth analysis. This analysis not only helps improve service quality but also effectively increases work efficiency. However, in practice, due to network bandwidth limitations, storage device capacity issues, or other technical constraints, some phone recordings combine the audio signals of both callers and save them in a mono format. While this mono storage method saves storage space and reduces transmission requirements, it poses significant challenges for subsequent audio processing and analysis. The limitations of mono recordings are particularly evident when performing advanced analytics such as speech recognition, character differentiation, and text conversion. It increases data processing complexity, reduces recognition accuracy, and may even result in the loss of critical information.

[0003] Therefore, how to optimize the storage and processing of telephone recordings is an urgent problem that needs to be solved. Summary of the Invention

[0004] The present application provides a recording channel segmentation method, device and storage medium to solve the problem of large limitations of mono recording.

[0005] In a first aspect, the present application provides a method for segmenting recording channels, comprising:

[0006] Obtaining a recording to be split, where the recording to be split is a mono recording and includes recording segments of at least two recording objects;

[0007] Extracting spectral features of the recording to be segmented from the recording to be segmented, and inputting the spectral features of the recording to be segmented into a trained recording segmentation model, wherein the recording segmentation model is used to segment the recording segments of at least two recording objects included in the recording to be segmented into recording segments of a single recording object;

[0008] Outputting arrangement information of the at least two recording objects and an embedded vector array through the recording segmentation model, the arrangement information including identification arrangement information, time arrangement information, and activity state arrangement information of the at least two recording objects in each time block, the embedded vector array being used to characterize recording segment features of different recording objects, and the number of embedded vectors in the embedded vector array being related to the number of recording objects included in the recording to be segmented;

[0009] Obtaining a non-silent time block according to the embedded vector array and the arrangement information in each time block;

[0010] generating a distance matrix according to the embedding vector array corresponding to the non-silent time block, wherein the distance matrix is ​​used to represent the distance between the embedding vectors in the embedding vector array corresponding to the non-silent time block;

[0011] Determining, based on the distance matrix and prior information related to the number of the recording objects, labeling information for each of the time blocks using a hierarchical clustering algorithm, wherein the labeling information is used to label the recording objects corresponding to the time blocks;

[0012] Generate an RTTM file corresponding to the recording to be segmented according to the annotation information, wherein the RTTM file includes a valid audio start time and an identifier of the recording object corresponding to the valid audio start time;

[0013] A channel-divided recording is generated according to the RTTM file, where different channels in the recording correspond to recording segments of different recording objects.

[0014] Optionally, the training process of the recording segmentation model includes:

[0015] Acquire a conversation dataset and a noise dataset, wherein the conversation dataset includes a timestamp and information of at least two recording objects;

[0016] Mixing the conversation dataset and the noise dataset to generate a noisy conversation dataset, wherein the noisy conversation dataset includes a first subset, a second subset, and a third subset, wherein the first subset is a training set of the recording segmentation model, the second subset is a validation set of the recording segmentation model, and the third subset is a test set of the recording segmentation model;

[0017] Extracting audio features from the noisy conversation dataset using a Mel filter to generate an audio feature set, which is used as input for the recording segmentation model;

[0018] The end-to-end recording object log model is trained using the audio feature set, the first subset, the second subset, and the third subset to generate the recording segmentation model.

[0019] Optionally, extracting the frequency spectrum features of the recording to be segmented from the recording to be segmented includes:

[0020] The recording to be segmented is subjected to frame processing, and each frame of the recording to be segmented is subjected to short-time Fourier transform to generate a frequency spectrum of the recording to be segmented;

[0021] Obtaining, according to the frequency spectrum of the recording to be segmented, Mel features, Mel cepstral coefficient features, and chroma features of the recording to be segmented;

[0022] Processing the recording to be segmented by using a psychoacoustic model to obtain texture features of the recording segments of the recording to be segmented;

[0023] Inputting the Mel feature, the Mel cepstral coefficient feature, the chroma feature, and the texture feature of the recording segment into an attention module to obtain an attention weight of each feature;

[0024] Generate a mel-weighted feature, a mel-cepstral coefficient weighted feature, a chroma weighted feature, and a recording segment texture weighted feature according to the mel feature, the mel-cepstral coefficient feature, the chroma feature, the recording segment texture feature, and the attention weight of each feature;

[0025] The mel-weighted feature, the mel-cephalometric coefficient weighted feature, the chroma weighted feature, and the recording segment texture weighted feature are fused and calculated to generate the spectrum feature of the recording to be segmented.

[0026] Optionally, the prior information is further related to the gender and / or age of the recording subject; and determining the labeling information of each time block by a hierarchical clustering algorithm based on the distance matrix and the prior information related to the number of the recording subjects includes:

[0027] Determine, according to the distance matrix, two first time blocks whose distance is less than a preset distance threshold;

[0028] If there is a significant conflict between the gender characteristics and / or age characteristics of the recording subjects corresponding to the two first time blocks, skipping the clustering of the two first time blocks;

[0029] If there is a significant conflict between the gender characteristics and / or age characteristics of the recording subjects corresponding to the two first time blocks, clustering and merging the two first time blocks;

[0030] updating the distance matrix according to the clustering merging results of the two first time blocks, and iterating the clustering process until the clustering result conforms to the prior information on the number of the recorded objects;

[0031] According to the clustering result, labeling information of each of the time blocks is determined.

[0032] Optionally, obtaining the non-silent time block according to the embedded vector array and the arrangement information in each time block includes:

[0033] Normalizing the embedding vector array to eliminate dimensional differences in the embedding vector array at different dimensions;

[0034] Preliminarily screening out, based on the activity status arrangement information in each arrangement information within each of the time blocks and the length of each of the time blocks, a second time block marked as "active" or "non-silent", wherein at least one recording object is in an active state within the second time block, and the length of the second time block is greater than or equal to a first length threshold;

[0035] If there are time blocks with overlapping areas in the second time blocks, processing the time blocks with overlapping areas according to the similarity of the embedded vector arrays corresponding to the overlapping areas of the time blocks with overlapping areas, the activity states and frequency information of the time blocks with overlapping areas to generate a first processing result, wherein the processing includes merging processing and discarding processing;

[0036] According to the first processing result, obtaining a third time block determined according to the corresponding second time block;

[0037] Calculating the Euclidean distance between the embedding vector arrays corresponding to the third time block by using similarity analysis to generate a second processing result, where the second processing result is used to characterize the similarity between the embedding vector arrays corresponding to the third time block;

[0038] If the second processing results corresponding to two adjacent third time blocks are less than or equal to the first similarity threshold, the two adjacent third time blocks are merged to generate the non-silent time block, and the length of the non-silent time block is greater than or equal to the third time block threshold.

[0039] Optionally, the training of an end-to-end recording object log model using the audio feature set, the first subset, the second subset, and the third subset to generate the recording segmentation model includes:

[0040] Construct an end-to-end recording object log model, wherein the input of the end-to-end recording object log model is the audio features of the recording to be segmented, and the output of the end-to-end recording object log model includes the arrangement information and the embedding vector array of the at least two recording objects; the end-to-end recording object log model includes a first convolutional layer before the encoder, an encoder, and a second convolutional layer after the encoder; the first convolutional layer before the encoder is a downsampling layer for scale fusion of input features, the encoder is used to vector encode the input features, and the second convolutional layer is used to map the vector-encoded input features to a high-dimensional space;

[0041] Training the end-to-end recording object log model for a preset number of rounds using the first subset, the second subset, and the third subset, and collecting a speech segmentation error rate DER and / or a loss function curve of each round of training;

[0042] If the speech segmentation error rate DER of each round of training, and / or the fluctuation of the loss function curve of each round of training is less than or equal to the preset fluctuation threshold, the training of the end-to-end recording object log model is terminated, and the recording segmentation model is generated according to the model parameters corresponding to the training results.

[0043] Optionally, the processing the recording to be segmented by using a psychoacoustic model to obtain texture features of the recording segments of the recording to be segmented includes:

[0044] Setting model parameters of the psychoacoustic model according to the recording to be segmented, the model parameters including bandwidth and frequency resolution of the filter;

[0045] Performing spectrum analysis on the recording to be segmented by fast Fourier transform to obtain frequency domain information of the recording to be segmented, and constructing a spectrum graph corresponding to the recording to be segmented based on the frequency domain information;

[0046] Acquire amplitude and phase information of each frequency component in the frequency domain information according to the spectrum diagram to generate target spectrum information;

[0047] Processing the target spectrum information using an auditory filter bank that simulates the response of the basilar membrane of the human ear to generate filtered spectrum data, wherein the filters in the auditory filter bank are used to simulate the perceptual sensitivity and resolution of the human ear to different frequencies;

[0048] Calculating the masking effect between the frequency components based on the filtered spectrum data to generate a masking threshold curve, wherein the masking threshold curve is used to represent the minimum sound level at which one sound can be masked by another sound at different frequencies;

[0049] Calculating psychoacoustic parameters of the recording to be segmented based on the masking threshold curve, wherein the psychoacoustic parameters include a loudness parameter and a sharpness parameter;

[0050] Extracting texture features of the initial recording segment of the recording to be segmented based on the filtered spectrum data, the masking threshold curve, and the psychoacoustic parameters, wherein the texture features of the initial recording segment include flatness, peak distribution, and harmonic structure of the target spectrum information;

[0051] The texture features of the initial recording segment are smoothed to generate texture features of the recording segment to be segmented, wherein the smoothing process is used to reduce the influence of noise and outliers in the texture features of the initial recording segment.

[0052] In a second aspect, the present application provides an electronic device, comprising: a processor, a communication interface, and a memory, wherein the processor is communicatively connected to the communication interface and the memory respectively;

[0053] The memory stores computer-executable instructions;

[0054] The communication interface communicates and interacts with external devices;

[0055] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0056] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0057] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which implements any one of the methods in the first aspect when executed by a processor.

[0058] The recording channel segmentation method, device and storage medium provided in the present application obtain a recording to be segmented, which is a mono recording, extract the spectral features of the recording to be segmented from the recording to be segmented, and input the spectral features of the recording to be segmented into a trained recording segmentation model, and output the arrangement information of the at least two recording objects and the embedding vector array through the recording segmentation model. According to the embedding vector array and the arrangement information within each time block, a non-silent time block is obtained, and a distance matrix is ​​generated according to the embedding vector array corresponding to the non-silent time block. According to the distance matrix and the prior information related to the number of the recording objects, the labeling information of each time block is determined by a hierarchical clustering algorithm. According to the labeling information, an RTTM file corresponding to the recording to be segmented is generated, and a recording after the channel segmentation is generated according to the RTTM file, so as to segment the mono recording into a multi-channel recording, thereby optimizing the storage and processing of telephone recordings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] Figure 1 A flowchart of a method for segmenting recording channels provided in an embodiment of the present application;

[0061] Figure 2 A flowchart of another recording channel segmentation method provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0063] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0065] The following describes in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 This is a flow chart of a method for segmenting recording channels provided in an embodiment of the present application. Figure 1 As shown, the method may include:

[0067] S101: Obtain a recording to be split, where the recording to be split is a mono recording.

[0068] The recording to be segmented includes recording segments of at least two recording objects.

[0069] The server retrieves the recording file to be segmented from a storage device or network. This file is mono, meaning all audio information is mixed in a single channel. The recording contains the conversation or voices of at least two recording subjects, and may be a recording of a multi-person conference or a phone conversation. The server must ensure that the recording file is in a supported format, such as WAV or MP3, and perform the necessary format conversion or decoding for subsequent processing.

[0070] For example, a company's internal project meeting is recorded, and the participants include project managers, developers, designers, etc. This recording file contains the voices of multiple recording objects, and the server needs to obtain this recording file as the recording to be segmented.

[0071] S102: extracting spectral features of the recording to be segmented from the recording to be segmented, and inputting the spectral features of the recording to be segmented into a trained recording segmentation model.

[0072] The recording segmentation model is used to segment the recording segments of at least two recording objects included in the recording to be segmented into recording segments of a single recording object.

[0073] The server uses signal processing technology to perform spectral analysis on the recordings to be segmented, extracting the spectral features of the recordings. These features may include Mel-Frequency Cepstral Coefficients (MFCCs) and chromaticity features, which reflect the time-frequency characteristics of the sound. Once extracted, these features are input into a pre-trained recording segmentation model. This model, trained using a large amount of labeled recording data, is capable of identifying and segmenting the sounds of different recording subjects.

[0074] Continuing with the example of a project meeting recording, the server will analyze the spectral characteristics of the recording file, such as the pitch and timbre of different people speaking, and input this information into the recording segmentation model so that the model can accurately identify the voices of different people.

[0075] S103. Output arrangement information of the at least two recording objects and an embedding vector array through the recording segmentation model.

[0076] The arrangement information includes identification arrangement information, time arrangement information, and activity status arrangement information of the at least two recording objects in each time block. The embedded vector array is used to characterize the recording segment features of different recording objects. The number of embedded vectors in the embedded vector array is related to the number of recording objects included in the recording to be segmented.

[0077] After processing, the recording segmentation model outputs information about the arrangement of recording objects within each time chunk, including their identities, timestamps, and activity status. The model also outputs an array of embedding vectors, each representing the sonic characteristics of a specific recording object. The number of embedding vectors matches the number of distinct sounds in the recording.

[0078] For example, when processing a meeting recording, the model might identify three primary voices: a project manager, a developer, and a designer. For each time block, the model outputs the presence and activity of these three individuals. Each individual's voice is also converted into an embedding vector for subsequent analysis and processing.

[0079] S104: Acquire a non-silent time block according to the embedded vector array and the arrangement information in each time block.

[0080] The server identifies time blocks containing valid sound (non-silence) based on the array of embedding vectors and the arrangement information within the time blocks. This usually involves clustering analysis of the embedding vectors and determining the activity state to exclude silence or background noise.

[0081] For example, in a meeting recording, there may be some silence, such as a break or when someone is thinking. The server identifies these silent sections and excludes them, retaining only those time blocks that contain actual speaking content.

[0082] S105: Generate a distance matrix according to the embedding vector array corresponding to the non-silent time block.

[0083] The distance matrix is ​​used to represent the distance between the embedded vectors in the embedded vector array corresponding to the non-silent time block.

[0084] For the embedding vectors corresponding to the filtered non-silent time blocks, the server calculates the distance between them and generates a distance matrix. This matrix reflects the similarity of sound features in different time blocks.

[0085] For example, when processing conference recordings, the server calculates the distance between the embedding vectors of the sounds in each non-silent time block. For example, if the sounds in two time blocks are very similar, then their corresponding values ​​in the distance matrix will be small.

[0086] S106 : Determine the labeling information of each of the time blocks using a hierarchical clustering algorithm according to the distance matrix and prior information related to the number of the recording objects.

[0087] The marking information is used to mark the recording object corresponding to the time block.

[0088] Using the distance matrix and prior information about the number of people in the recording (e.g., how many people are speaking in the recording), the server uses a hierarchical clustering algorithm to determine the labels for each time block. These labels indicate which person's voice corresponds to each time block.

[0089] For example, in a conference recording, the server uses a clustering algorithm based on the distance matrix and prior knowledge (e.g., three people speaking in a meeting) to determine who was speaking in each time block. This way, each time block is labeled with the corresponding speaker's information.

[0090] S107: Generate an RTTM file corresponding to the recording to be segmented according to the annotation information.

[0091] The RTTM file includes the valid audio start time and the identifier of the recording object corresponding to the valid audio start time.

[0092] Based on the annotation information, the server generates a Rich Transcription TimeMarked (RTTM) file. This file contains the start time of the active audio and the identifier of the recording object at that time. RTTM files are a commonly used annotation format for speech recognition and speech segmentation tasks.

[0093] For example, for a conference recording, the server generates an RTTM file that details the start and end time of each person's speech. This file can be used for subsequent tasks such as speech recognition and speech transcription.

[0094] S108: Generate a channel-segmented recording according to the RTTM file.

[0095] Wherein, different channels in the recording correspond to recording segments of different recording objects.

[0096] The server will split the original mono recording into multiple channels based on the RTTM file, with each channel corresponding to the sound of a recording object. In this way, the originally mixed sounds are successfully separated.

[0097] For example, when recording a meeting, the server will eventually generate a recording file with three channels, one for each of the project manager, developer, and designer, allowing users to listen to each person's speech separately.

[0098] The method provided by the embodiment of the present application obtains a recording to be segmented, which is a monophonic recording, extracts the spectral features of the recording to be segmented from the recording to be segmented, and inputs the spectral features of the recording to be segmented into a trained recording segmentation model, and outputs the arrangement information of the at least two recording objects and an embedding vector array through the recording segmentation model. According to the embedding vector array and the arrangement information within each time block, a non-silent time block is obtained, and a distance matrix is ​​generated according to the embedding vector array corresponding to the non-silent time block. According to the distance matrix and the prior information related to the number of the recording objects, the labeling information of each time block is determined by a hierarchical clustering algorithm. According to the labeling information, an RTTM file corresponding to the recording to be segmented is generated, and a recording after channel segmentation is generated according to the RTTM file, so as to segment the monophonic recording into a multi-channel recording, thereby optimizing the storage and processing of telephone recordings.

[0099] In a possible implementation, the aforementioned step S102 may be implemented through the following sub-steps:

[0100] S1021 , dividing the recording to be segmented into frames, and performing short-time Fourier transform on each frame of the recording to be segmented to generate a frequency spectrum of the recording to be segmented.

[0101] The server first reads the recording file to be segmented into memory. Then, the server will segment the recording into several short frames according to the predetermined frame length and frame shift. Usually the frame length is selected to be 20-40 milliseconds, and the frame shift is about half of the frame length. This ensures that there is a certain overlap between adjacent frames to improve the accuracy of the analysis. After the framing is completed, the server will perform a short-time Fourier transform (STFT) on each frame of data, which is a method of converting signals between the time domain and the frequency domain. Through STFT, the server can obtain the spectrum of each frame of data, which reflects the energy distribution of the sound signal at different frequencies.

[0102] For example, the server reads a 10-second audio file to be segmented and decides to use a 30-millisecond frame length and a 15-millisecond frame shift for the segmentation process. This results in approximately 666 frames. The server then performs a short-time Fourier transform on each frame, generating 666 spectra. Each spectrum details the energy distribution of the sound signal at different frequencies within that frame.

[0103] S1022. Obtain, based on the frequency spectrum of the recording to be segmented, Mel features, Mel cepstral coefficient features, and chroma features of the recording to be segmented.

[0104] The server uses the spectrum data previously obtained through STFT to further extract Mel-frequency cepstral coefficients (MFCCs), Mel-frequency cepstral coefficients (MFCCs), and chroma features. Mel-frequency features are designed based on the characteristics of the human auditory system and can simulate the human ear's sensitivity to sounds of different frequencies. Chroma features, which represent musical pitch, are highly descriptive of the melodic and harmonic information of musical signals.

[0105] For example, the server applies a specific algorithm to each spectral data point to extract 13-dimensional Mel features, 13-dimensional Mel first-order difference features, and 13-dimensional Mel second-order difference features, which together constitute a 39-dimensional Mel feature vector. At the same time, the server also extracts 12-dimensional chromaticity features, which capture the musical structure and melodic information in the recording.

[0106] S1023: Process the recording to be segmented using a psychoacoustic model to obtain texture features of the recording segments of the recording to be segmented.

[0107] The server uses a psychoacoustic model to process the recordings to be segmented. This model simulates the perceptual characteristics of the human auditory system and can predict the perceived quality of an audio signal based on its physical properties. The server uses this model to analyze the sound quality of each part of the recording, such as clarity and noise level, to determine the texture characteristics of the recording segments.

[0108] The server inputs the recording to be segmented into the psychoacoustic model, and the model outputs the sound quality evaluation results of each part of the recording, including the clarity index, noise level, etc. These evaluation results constitute the texture characteristics of the recording segment.

[0109] Specifically, this step can be implemented through the following sub-steps:

[0110] S1. Setting model parameters of the psychoacoustic model according to the recording to be segmented.

[0111] The model parameters include the bandwidth and frequency resolution of the filter.

[0112] In audio processing, psychoacoustic models are a key tool, simulating how humans perceive sounds of different frequencies. Before performing audio segmentation, the server first needs to set the parameters of the psychoacoustic model based on the recording to be segmented. These parameters primarily include the filter bandwidth and frequency resolution. The filter bandwidth determines which frequency range of sound will be captured and processed by the filter, while the frequency resolution affects the ability to capture the details of the sound signal. The server will fine-tune these parameters based on the characteristics of the human auditory system and the specific content and quality of the recording to be processed. For example, if the recording contains rich high-frequency information, the server may increase the filter bandwidth in the high-frequency band to improve the processing ability of sounds in this frequency band.

[0113] S2. Performing spectrum analysis on the recording to be segmented by fast Fourier transform to obtain frequency domain information of the recording to be segmented, and constructing a spectrum graph corresponding to the recording to be segmented according to the frequency domain information.

[0114] The server uses a Fast Fourier Transform (FFT) to perform spectral analysis on the segmented recordings. The FFT is a highly efficient algorithm that converts time-domain signals into frequency-domain signals, revealing the strength of different frequency components in the audio. After performing the FFT, the server constructs a spectrogram of the recording based on the obtained frequency-domain information. This graph visually displays the energy distribution of the audio at each frequency.

[0115] S3. Acquire amplitude and phase information of each frequency component in the frequency domain information according to the spectrum diagram to generate target spectrum information.

[0116] The server performs an in-depth analysis of the spectrum data obtained through the FFT conversion. It carefully examines the peaks in the spectrogram, which represent the strength of different frequency components in the audio. For each peak, the server records the corresponding frequency, amplitude, and phase information. This information is the core characteristic of the audio signal and can reflect the basic structure and timbre of the audio. The server integrates this data to form the target spectrum information.

[0117] S4. Processing the target spectrum information using an auditory filter bank that simulates the response of the basilar membrane of the human ear to generate filtered spectrum data, wherein the filters in the auditory filter bank are used to simulate the perception sensitivity and resolution of the human ear to different frequencies.

[0118] The server processes the target spectrum information obtained in the previous step using a set of specially designed auditory filters. These filters are designed to mimic the human ear's perception of sounds of different frequencies, simulating the human ear's sensitivity and resolution to different frequencies. When the target spectrum information passes through these filters, some frequency components may be enhanced while others may be suppressed, depending on the filter characteristics. Ultimately, the server obtains a set of filtered spectrum data that more closely resembles the human ear's auditory perception.

[0119] S5. Calculate the masking effect between the frequency components based on the filtered spectrum data, and generate a masking threshold curve.

[0120] The masking threshold curve is used to represent the minimum sound level at which one sound can be masked by another sound at different frequencies.

[0121] Based on the filtered spectrum data, the server analyzes the relationships between different frequency components, calculates which frequencies mask other frequencies, and determines the masking thresholds. These thresholds are plotted as a curve, the masking threshold curve, which clearly shows the minimum sound level at which one sound can be masked by another at different frequencies.

[0122] S6. Obtain psychoacoustic parameters of the recording to be segmented based on the masking threshold curve, where the psychoacoustic parameters include a loudness parameter and a sharpness parameter.

[0123] The server further uses the masking threshold curve to calculate the psychoacoustic parameters of the audio recording to be segmented. These parameters, such as loudness and sharpness, quantitatively describe the auditory perception of the audio signal. For example, loudness reflects the intensity of a sound, while sharpness describes its harshness. These psychoacoustic parameters provide important reference for subsequent feature extraction and audio segmentation.

[0124] S7. Extracting texture features of the initial recording segment of the recording to be segmented based on the filtered spectrum data, the masking threshold curve, and the psychoacoustic parameters.

[0125] The texture features of the initial recording segment include the flatness, peak distribution, and harmonic structure of the target spectrum information.

[0126] The server begins extracting texture features from the initial recording. These features are primarily based on filtered spectral data, masking threshold curves, and psychoacoustic parameters. The server analyzes the spectral flatness—that is, whether the frequency components are evenly distributed. It also examines the peak distribution to determine which frequency components have high energy. Furthermore, it examines the harmonic structure—the relationship between the fundamental frequency and harmonics of the sound. Together, these features constitute the texture features of the initial recording, reflecting the uniqueness and recognizability of the audio.

[0127] S8. Smoothing the texture features of the initial recording segment to generate texture features of the recording segment to be segmented.

[0128] The smoothing process is used to reduce the influence of noise and outliers in the texture features of the initial recording segment.

[0129] The server smoothes the texture features of the initial recording segment. This is because actual audio signals may contain noise and outliers, which can affect the accuracy and stability of the features. Through smoothing, the server can reduce the impact of these adverse factors, making the features more reliable. After smoothing, the server ultimately generates the texture features of the recording segment to be segmented.

[0130] S1024: Input the Mel feature, the Mel cepstral coefficient feature, the chroma feature, and the recording segment texture feature into an attention module to obtain an attention weight of each feature.

[0131] The server feeds these features into a pre-trained attention module. The attention module is a deep learning model that learns and automatically assigns different attention weights to each feature. These weights reflect the importance of different features for subsequent tasks, such as audio segmentation.

[0132] For example, the server has already compiled the mel features, mel cepstral coefficient features, chroma features, and audio segment texture features extracted in the previous step. It then inputs these feature vectors into the attention module. After internal calculations within the model, the attention module assigns a weight of 0.4 to the mel features, 0.3 to the mel cepstral coefficient features, 0.2 to the chroma features, and 0.1 to the audio segment texture features. These weights are used in subsequent weighted feature generation.

[0133] S1025. Generate mel-weighted features, mel-cepstral coefficient weighted features, chroma weighted features, and recording segment texture weighted features based on the mel features, the mel-cepstral coefficient features, the chroma features, the recording segment texture features, and the attention weights of the features.

[0134] The server uses the attention weights obtained in step S1024 to weight the mel features, mel cepstral coefficient features, chroma features, and recording segment texture features. The purpose of weighting is to emphasize the features that are most important to the segmentation task while reducing the impact of unimportant features. The server generates mel-weighted features, mel cepstral coefficient weighted features, chroma weighted features, and recording segment texture weighted features by multiplying the value of each feature by the corresponding weight.

[0135] For example, based on the attention weights obtained in step S1024, the server begins weighting each feature. Mel-value features are multiplied by 0.4, Mel-value cepstral coefficient features are multiplied by 0.3, chroma features are multiplied by 0.2, and recording texture features are multiplied by 0.1. These weighted features better reflect the key audio information of the recording to be segmented, providing more accurate feature input for subsequent segmentation tasks.

[0136] S1026 , fusing and calculating the Mel-weighted feature, the Mel-cepstral coefficient weighted feature, the chroma weighted feature, and the recording segment texture weighted feature to generate a spectrum feature of the recording to be segmented.

[0137] The server then performs feature fusion calculations. This step aims to fuse Mel-weighted features, Mel-weighted cepstral coefficient features, chroma-weighted features, and texture-weighted features of the recording segment into a comprehensive spectral feature. The server can employ various fusion strategies, such as simple feature concatenation and weighted averaging, or more advanced fusion methods, such as feature fusion via deep learning models. The fused features contain comprehensive information about the recording to be segmented, providing strong support for subsequent audio segmentation.

[0138] For example, the server uses a deep learning model for feature fusion. It takes Mel-weighted features, Mel-weighted cepstral coefficient features, chroma-weighted features, and texture-weighted features of the recording segment as input. Through complex computations, the model effectively fuses these features. Ultimately, the server generates a comprehensive spectral feature that contains comprehensive information about the recording to be segmented. This feature is directly used in subsequent audio segmentation algorithms to improve segmentation accuracy and efficiency.

[0139] The method provided in the embodiment of the present application is to process the recording to be segmented by dividing it into frames and performing a short-time Fourier transform on each frame of the recording to be segmented to generate the spectrum of the recording to be segmented, and obtain the Mel feature, Mel cepstral coefficient feature, and chroma feature of the recording to be segmented based on the spectrum of the recording to be segmented. The recording to be segmented is processed by a psychoacoustic model to obtain the texture feature of the recording segment of the recording to be segmented, and the Mel feature, the Mel cepstral coefficient feature, the chroma feature, and the recording segment texture feature are input into the attention module to obtain the attention weight of each feature. Based on the Mel feature, the Mel cepstral coefficient feature, the chroma feature, the recording segment texture feature, and the attention weight of each feature, a Mel-weighted feature, a Mel cepstral coefficient weighted feature, a chroma weighted feature, and a recording segment texture weighted feature are generated. The mel-weighted features, the mel-cephalometric weighted features, the chroma-weighted features, and the recording segment texture-weighted features are fused and calculated to generate the spectral features of the recording to be segmented, thereby improving the accuracy and efficiency of extracting the spectral features of the recording to be segmented from the recording to be segmented.

[0140] In a possible implementation, the aforementioned step S104 may be implemented through the following sub-steps:

[0141] S1041. Normalize the embedded vector array to eliminate dimensional differences of the embedded vector array in different dimensions.

[0142] After receiving the embedded vector array, the server first performs normalization. This is because the dimensions in the embedded vector array may have different dimensions and value ranges, and direct comparisons or calculations can distort the results. Normalization scales the data across dimensions to the same magnitude, typically by mapping the data to the range [0, 1] or [-1, 1]. This eliminates dimensional differences across dimensions, making subsequent calculations more accurate. The server uses methods such as min-max normalization to perform a linear transformation on the data in each dimension to achieve normalization.

[0143] For example, suppose the server receives a three-dimensional embedding vector array with different ranges for each dimension. To eliminate this dimensionality difference, the server normalizes the data in each dimension. For example, if the original data for the first dimension is in the range [10, 100], the server will map it to the range [0, 1]. Similar processing is performed for the second and third dimensions. This way, the data in each dimension of the processed embedding vector array has the same magnitude, facilitating subsequent analysis and calculations.

[0144] S1042: Preliminarily filter out second time blocks marked as “active” or “non-silent” based on the activity status arrangement information in each arrangement information within each time block and the length of each time block.

[0145] At least one recording object is in an active state within the second time block, and the length of the second time block is greater than or equal to a first length threshold.

[0146] The server will perform a preliminary screening based on the activity status arrangement information within each time block and the length of the time block. The activity status arrangement information can be a mark indicating whether the recording subject is in an active state (such as a speaking state) as identified by voice recognition technology. The server will filter out time blocks in which at least one recording subject is active and the time block length is greater than or equal to a preset first length threshold, and mark these time blocks as "second time blocks." This is done to eliminate time periods that are too short or without valid voice activity, thereby improving the efficiency and accuracy of subsequent processing.

[0147] For example, suppose the server is processing a conference recording containing multiple time blocks. The server first checks the activity status arrangement information within each time block to identify time periods in which the recorded subject is speaking. The server then further filters out time blocks within these time blocks whose length is greater than or equal to a first length threshold (e.g., 5 seconds) as "second time blocks." This way, the server only focuses on time periods of sufficient length and containing valid voice activity.

[0148] S1043. If there is a time block with an overlapping area in the second time block, the time block with the overlapping area is processed according to the similarity of the embedded vector array corresponding to the overlapping area of ​​the time block with the overlapping area, the activity state and frequency information of the time block with the overlapping area to generate a first processing result.

[0149] The processing includes merging processing and selecting processing.

[0150] When the server discovers overlapping regions between the second time chunks, it processes these time chunks based on the similarity, activity status, and frequency information of the embedding vector arrays corresponding to the overlapping regions. If the embedding vector arrays corresponding to the overlapping regions are very similar, and the activity status and frequency information are consistent, the server may choose to merge these time chunks. If the similarity is not high or there is a conflict in the activity status or frequency information, the server may choose to discard one time chunk while retaining the other. After processing is complete, the server generates a first processing result, which reflects the status of the processed time chunks.

[0151] For example, suppose there are two second time blocks A and B, and there is an overlapping area between them. The server will compare the similarity of the embedding vector arrays corresponding to the overlapping area, while taking into account the activity status and frequency information in A and B. If the embedding vector arrays in the overlapping area are very similar, and the recording objects in A and B are both active, and the frequency information is similar, then the server may decide to merge the two time blocks A and B. On the contrary, if the similarity is not high or there is other conflicting information, the server may choose to retain one of the time blocks (such as A) and discard the other (such as B). The result of this processing is output as the first processing result.

[0152] S1044. According to the first processing result, obtain a third time block determined according to the second time block.

[0153] Based on the first processing results (i.e., the time blocks after merging or discarding), the server further determines third time blocks. These third time blocks are derived from the processing results of the second time blocks and represent the time periods determined to contain valid voice information after preliminary screening and processing. These third time blocks will serve as the basis for subsequent similarity analysis and merging operations.

[0154] For example, if the server decides to merge time chunks A and B, the resulting merged time chunk can be used as a third time chunk. Similarly, if the server chooses to keep time chunk A and discard time chunk B, time chunk A will also be selected as a third time chunk. These third time chunks will be used for subsequent similarity analysis and possible merging operations.

[0155] S1045: Calculate the Euclidean distance between the embedded vector arrays corresponding to the third time block by using similarity analysis to generate a second processing result.

[0156] The second processing result is used to characterize the similarity between the embedding vector arrays corresponding to the third time block.

[0157] The server can use similarity analysis methods (such as calculating Euclidean distance) to evaluate the similarity between the embedding vector arrays corresponding to the third time block. Euclidean distance is a commonly used distance metric that reflects the straight-line distance between two vectors in multidimensional space. By calculating the Euclidean distance between the embedding vectors corresponding to each third time block, the server can obtain a second processing result that represents the degree of similarity between these embedding vectors.

[0158] For example, suppose the server has two third-time chunks, C and D, each corresponding to two sets of embedding vector arrays. To assess the similarity between these two sets of embedding vectors, the server calculates the Euclidean distance between them. The smaller this distance, the more similar the embedding vectors corresponding to C and D are; conversely, the greater the difference. This calculation result is output as the second processing result.

[0159] S1046: If the second processing results corresponding to two adjacent third time blocks are less than or equal to the first similarity threshold, merge the two adjacent third time blocks to generate the non-silent time block.

[0160] The length of the non-silent time block is greater than or equal to a third time block threshold.

[0161] Based on the second processing result (i.e., the similarity evaluation between the embedding vectors corresponding to each third time block), the server will determine which adjacent third time blocks have a high degree of similarity. If the second processing results (such as the Euclidean distance) corresponding to two adjacent third time blocks are less than or equal to the preset first similarity threshold, the server will merge the two time blocks into a larger time block, called a "non-silent time block." This non-silent time block represents a continuous and valid voice activity time period in the recording. At the same time, in order to ensure that the merged time block has sufficient length and information volume, the server will also check whether the length of the merged non-silent time block is greater than or equal to the preset third time block threshold.

[0162] For example, if the server calculates that the Euclidean distance between the third time blocks C and D is less than or equal to the first similarity threshold (e.g., a predetermined small value), it will decide to merge the two time blocks C and D to form a new non-silent time block. This new time block will contain all the speech information in C and D, and its length will be greater than or equal to the third time block threshold (e.g., 10 seconds). This processed non-silent time block will be more convenient for subsequent analysis and utilization.

[0163] The method provided in the embodiment of the present application improves the availability of the generated non-silent time blocks, thereby improving the accuracy of recording segmentation.

[0164] In a possible implementation, the prior information is also related to the gender and / or age of the recording subject. The aforementioned step S106 can be implemented by the following sub-steps:

[0165] S1061: Determine, according to the distance matrix, two first time blocks whose distance is less than a preset distance threshold.

[0166] The server traverses the entire distance matrix, where each element represents the similarity or distance between different time chunks. The server checks each element in the matrix and finds time chunk pairs whose distance value is less than a preset distance threshold. These time chunk pairs are considered similar in characteristics and may be from the same recording subject. For example, if the distance value between time chunk A and time chunk B is less than the threshold, then A and B are considered a pair of "similar" time chunks.

[0167] For example, suppose the server has a 4x4 distance matrix representing the distances between each of four time chunks. The server sets a distance threshold of 0.5. While traversing the matrix, the server discovers that the distance between time chunk 1 and time chunk 3 is 0.4, which is less than the threshold. Therefore, the server determines that time chunk 1 and time chunk 3 are the "first time chunk" pair whose distance is less than the preset threshold.

[0168] S1062: If there is a significant conflict between the gender characteristics and / or age characteristics of the recording objects corresponding to the two first time blocks, skip clustering of the two first time blocks.

[0169] After identifying similar time blocks, the server can further examine the gender and age characteristics of the recording subjects corresponding to these time blocks. If there is a clear inconsistency in the gender or age characteristics of two time blocks, such as one voice is clearly male and the other is female, or one voice is clearly a child and the other is an adult, then the two time blocks will not be clustered together.

[0170] For example, when checking the gender features of time blocks 1 and 3, the server finds that the recording of time block 1 is a male voice, while the recording of time block 3 is a female voice. Due to the significant conflict in gender features, the server decides to skip clustering of time blocks 1 and 3.

[0171] S1063: If there is a significant conflict between the gender characteristics and / or age characteristics of the recording subjects corresponding to the two first time blocks, cluster and merge the two first time blocks.

[0172] If two similar time blocks have no obvious conflicts in gender and age characteristics, the server will cluster and merge them, that is, they are considered to come from the same recording subject.

[0173] For example, suppose the server finds that the distance between time chunks 2 and 4 in the distance matrix is ​​less than a threshold, and that their gender and age characteristics match, indicating they are both adult male voices. Therefore, the server will cluster and merge time chunks 2 and 4, treating them as recordings of the same subject.

[0174] S1064. Update the distance matrix according to the clustering merging result of the two first time blocks, and iterate the clustering process until the clustering result meets the prior information on the number of the recorded objects.

[0175] Whenever two time chunks are clustered and merged, the server updates the distance matrix, treating the merged time chunk as a new unit and recalculating the distances between it and other time chunks. This process continues iteratively until the number of clusters matches the number of known recording objects.

[0176] For example, after clustering time chunks 2 and 4, the server recalculates the distance between the new time chunk (time chunk 2 + 4) and time chunks 1 and 3, and updates the distance matrix. This process continues until the number of clusters matches the number of recordings known to the server.

[0177] S1065: Determine labeling information for each of the time blocks according to the clustering result.

[0178] Once the clustering process is complete, the server will assign annotation information to each time block based on the final clustering results. This annotation information may include the identity, gender, age, etc. of the recording subject, which facilitates subsequent data analysis and processing.

[0179] For example, suppose the final clustering result is three clusters, corresponding to three different recording objects. The server will assign a unique identifier to each cluster, such as "Recording Object 1," "Recording Object 2," and "Recording Object 3." The server will then attach these identifiers as annotation information to the corresponding time blocks for rapid identification and retrieval.

[0180] The method provided in the embodiment of the present application determines, based on the distance matrix, two first time blocks whose distance is less than a preset distance threshold. If there is a significant conflict in the gender characteristics and / or age characteristics of the recording objects corresponding to the two first time blocks, the clustering behavior of the two first time blocks is skipped; if there is a significant conflict in the gender characteristics and / or age characteristics of the recording objects corresponding to the two first time blocks, the two first time blocks are clustered and merged. Based on the clustering and merging results of the two first time blocks, the distance matrix is ​​updated, and the clustering process is iterated until the clustering results meet the prior information on the number of the recording objects. Based on the clustering results, the labeling information of each of the time blocks is determined. Thus, by adding a comprehensive analysis of the identity, gender, and age of the recording objects, the accuracy of the segmented recordings is further improved.

[0181] The following is a detailed introduction to the training process of the recording segmentation model. Figure 2This is a flow chart of another method for dividing recording channels provided in an embodiment of the present application. Figure 2 As shown, the method may include:

[0182] S201. Acquire a conversation dataset and a noise dataset, wherein the conversation dataset includes a timestamp and information of at least two recording objects.

[0183] The server can obtain conversation and noise datasets from public data sources or proprietary databases. The conversation dataset contains conversations between multiple recordings. Each conversation is timestamped and includes information about at least two recordings, including the recordings' identities, gender, and ages. The noise dataset contains various environmental noises, such as traffic, wind, and machine noise, to simulate interference factors found in real environments.

[0184] For example, the server downloaded a dataset of conversations in multiple languages, including English and Chinese, from a large multilingual conversation database. These conversations consist of two or more participants. The server also obtained a dataset of various noise types, including street noise and office murmurs, from a sound library. This data will provide rich material for subsequent model training.

[0185] S202: Mix the conversation dataset and the noise dataset to generate a noise conversation dataset.

[0186] The noisy conversation dataset includes a first subset, a second subset, and a third subset. The first subset is the training set of the recording segmentation model, the second subset is the validation set of the recording segmentation model, and the third subset is the test set of the recording segmentation model.

[0187] The server mixes the conversation dataset with the noise dataset to simulate real-world conversation scenarios. During the mixing process, the server superimposes the noise data onto the conversation data according to a preset ratio, generating noisy conversation data. This mixed data is then divided into three subsets: the first subset serves as the training set for training the audio segmentation model; the second subset serves as the validation set for verifying the model's performance during training; and the third subset serves as the test set for testing the model's performance after training.

[0188] For example, the server uses professional audio editing software to mix noise data such as street noise and office murmurs with conversation data. This mixed data is more realistic and helps improve the model's generalization capabilities. This data is evenly distributed across training, validation, and test sets to ensure the model performs well in a variety of noise environments.

[0189] S203. Extract audio features from the noisy conversation dataset using a Mel filter to generate an audio feature set, which is used as input for the recording segmentation model.

[0190] The server uses a Mel filter to extract features from the mixed noisy conversation dataset. Mel filters are a filter bank designed based on the characteristics of the human auditory system, effectively extracting features from audio signals. The audio data processed by the Mel filter is converted into a series of feature vectors, which serve as input to the recording segmentation model.

[0191] For example, the server can apply a Mel filter to the mixed conversation data to convert it into a Mel spectrogram. These spectrograms can clearly show the frequency components and energy distribution in the audio signal, providing strong feature support for subsequent model training.

[0192] S204: Train an end-to-end recording object log model using the audio feature set, the first subset, the second subset, and the third subset to generate the recording segmentation model.

[0193] The server uses the audio feature set and three previously divided subsets (training set, validation set, and test set) to train an end-to-end recording object log model. This model directly outputs the arrangement information and embedding vector array of the recording objects, thereby achieving accurate segmentation of the recordings. During training, the server continuously adjusts the model parameters to optimize its performance and monitors the training progress and results using the validation set. When the model's performance on the validation set meets the preset standard, the server stops training and saves the model parameters.

[0194] Specifically, this step can be implemented through the following sub-steps:

[0195] S2041. Build an end-to-end recording object log model.

[0196] Among them, the input of the end-to-end recording object log model is the audio features of the recording to be segmented, and the output of the end-to-end recording object log model includes the arrangement information and embedded vector array of the at least two recording objects; the end-to-end recording object log model includes a first convolutional layer before the encoder, an encoder, and a second convolutional layer after the encoder; the first convolutional layer before the encoder is a downsampling layer for scale fusion of input features, the encoder is used to vector encode the input features, and the second convolutional layer is used to map the vector-encoded input features to a high-dimensional space.

[0197] When building an end-to-end recording object log model, we first need to determine the model's input and output formats. The input is the audio features of the recording to be segmented. These features can be MFCCs (Mel-Frequency Cepstral Coefficients) extracted using an acoustic model or other types of audio features. The output includes the arrangement information and embedding vector arrays of at least two recording objects, which are crucial for subsequent speech segmentation.

[0198] The model construction begins with the first convolutional layer before the encoder, which is primarily used to downsample the input features for scale fusion. Downsampling aims to reduce the dimensionality of features while retaining key information for subsequent encoder processing. The encoder is responsible for encoding these features into vectors that capture the key characteristics of the recorded subject. The encoder can use structures such as the Transformer or LSTM to efficiently process sequential data.

[0199] The second convolutional layer after the encoder maps the encoder output vector into a high-dimensional space. This step helps the model better learn and distinguish different recording objects. Ultimately, the model outputs an array of embedded vectors and the arrangement of the recording objects, which serves as the basis for subsequent speech segmentation.

[0200] S2042. Train the end-to-end recording object log model for a preset number of rounds using the first subset, the second subset, and the third subset, and collect the speech segmentation error rate DER and / or the loss function curve of each round of training.

[0201] When training the end-to-end audio recording object logging model, we use three different datasets: Subset 1, Subset 2, and Subset 3. These datasets contain recordings from different scenarios and conditions to ensure the model's generalization capabilities. During training, we set a preset number of epochs, such as 100, and allow the model to iteratively learn on these datasets.

[0202] After each round of training, we collect the speech segmentation error rate (DER) and loss function curve. DER is an important evaluation metric used to measure the model's performance on speech segmentation tasks. The loss function curve reflects the model's learning progress and convergence trend during training.

[0203] For example, suppose we train the model for 100 epochs using three subsets. After each epoch, we record the current DER and loss function values. As the number of training epochs increases, we can observe that the DER and loss function values ​​gradually decrease, indicating that the model is gradually learning how to segment speech more accurately.

[0204] S2043. If the speech segmentation error rate DER of each round of training, and / or the fluctuation of the loss function curve of each round of training is less than or equal to the preset fluctuation threshold, the training of the end-to-end recording object log model is terminated, and the recording segmentation model is generated according to the model parameters corresponding to the training results.

[0205] During training, we continuously monitor the changes in the DER and loss function curves for each round of training. When the fluctuations in these indicators are less than or equal to our preset fluctuation thresholds, we consider the model to have converged and reached a relatively stable state, and training can be terminated.

[0206] Once training is complete, we generate the final audio segmentation model based on the model parameters corresponding to the training results. This model will be able to accurately segment different objects in the audio recording, providing a foundation for subsequent tasks such as speech recognition and speaker identification.

[0207] For example, suppose that during the 80th epoch of training, we find that the fluctuations in both the DER and loss function curves are below a preset fluctuation threshold. At this point, we decide to terminate model training and generate a final audio segmentation model based on the current model parameters. This model demonstrates good performance in real-world applications, accurately segmenting different speakers in audio recordings.

[0208] The method provided in an embodiment of the present application obtains a conversation dataset and a noise dataset, wherein the conversation dataset includes a timestamp and information about at least two recording objects. The conversation dataset and the noise dataset are mixed to generate a noisy conversation dataset. Audio features in the noisy conversation dataset are extracted using a Mel filter to generate an audio feature set, which is used as input for the recording segmentation model. The end-to-end recording object log model is trained using the audio feature set, the first subset, the second subset, and the third subset to generate the recording segmentation model. By training the recording segmentation model, the accuracy and efficiency of subsequent segmentation of the recording to be segmented based on the recording segmentation model are improved.

[0209] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device is used to execute the aforementioned recording channel segmentation method, for example, it can be the aforementioned server. Figure 3 As shown, the electronic device 300 may include: at least one processor 301 , a memory 302 , and a communication interface 303 .

[0210] The memory 302 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.

[0211] The memory 302 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0212] The processor 301 is configured to execute computer-executable instructions stored in the memory 302 to implement the method described in the aforementioned method embodiment. The processor 301 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0213] The processor 301 can communicate and interact with an external device through the communication interface 303. The external device can be, for example, a storage device that stores the recording to be segmented. In a specific implementation, if the communication interface 303, the memory 302, and the processor 301 are implemented independently, the communication interface 303, the memory 302, and the processor 301 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc., but this does not mean that there is only one bus or one type of bus.

[0214] Optionally, in a specific implementation, if the communication interface 303, the memory 302 and the processor 301 are integrated on a chip, the communication interface 303, the memory 302 and the processor 301 can complete communication through an internal interface.

[0215] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above embodiments.

[0216] The present application also provides a program product, the program product including execution instructions stored in a readable storage medium. At least one processor of a computing device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions so that the computing device implements the above-mentioned recording channel segmentation method.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for segmenting recording channels, characterized in that: The method comprises: Obtaining a recording to be split, where the recording to be split is a mono recording and includes recording segments of at least two recording objects; Extracting spectral features of the recording to be segmented from the recording to be segmented, and inputting the spectral features of the recording to be segmented into a trained recording segmentation model, wherein the recording segmentation model is used to segment the recording segments of at least two recording objects included in the recording to be segmented into recording segments of a single recording object; Outputting arrangement information of the at least two recording objects and an embedded vector array through the recording segmentation model, the arrangement information including identification arrangement information, time arrangement information, and activity state arrangement information of the at least two recording objects in each time block, the embedded vector array being used to characterize recording segment features of different recording objects, and the number of embedded vectors in the embedded vector array being related to the number of recording objects included in the recording to be segmented; Obtaining a non-silent time block according to the embedded vector array and the arrangement information in each time block; generating a distance matrix according to the embedding vector array corresponding to the non-silent time block, wherein the distance matrix is ​​used to represent the distance between the embedding vectors in the embedding vector array corresponding to the non-silent time block; Determining, based on the distance matrix and prior information related to the number of the recording objects, labeling information for each of the time blocks using a hierarchical clustering algorithm, wherein the labeling information is used to label the recording objects corresponding to the time blocks; Generate an RTTM file corresponding to the recording to be segmented according to the annotation information, wherein the RTTM file includes a valid audio start time and an identifier of the recording object corresponding to the valid audio start time; A channel-divided recording is generated according to the RTTM file, where different channels in the recording correspond to recording segments of different recording objects.

2. The method according to claim 1, characterized in that The training process of the recording segmentation model includes: Acquire a conversation dataset and a noise dataset, wherein the conversation dataset includes a timestamp and information of at least two recording objects; Mixing the conversation dataset and the noise dataset to generate a noisy conversation dataset, wherein the noisy conversation dataset includes a first subset, a second subset, and a third subset, wherein the first subset is a training set of the recording segmentation model, the second subset is a validation set of the recording segmentation model, and the third subset is a test set of the recording segmentation model; Extracting audio features from the noisy conversation dataset using a Mel filter to generate an audio feature set, which is used as input for the recording segmentation model; The end-to-end recording object log model is trained using the audio feature set, the first subset, the second subset, and the third subset to generate the recording segmentation model.

3. The method according to claim 1, characterized in that The extracting the frequency spectrum features of the recording to be segmented from the recording to be segmented includes: The recording to be segmented is subjected to frame processing, and each frame of the recording to be segmented is subjected to short-time Fourier transform to generate a frequency spectrum of the recording to be segmented; Obtaining, according to the frequency spectrum of the recording to be segmented, Mel features, Mel cepstral coefficient features, and chroma features of the recording to be segmented; Processing the recording to be segmented by using a psychoacoustic model to obtain texture features of the recording segments of the recording to be segmented; Inputting the Mel feature, the Mel cepstral coefficient feature, the chroma feature, and the texture feature of the recording segment into an attention module to obtain an attention weight of each feature; Generate a mel-weighted feature, a mel-cepstral coefficient weighted feature, a chroma weighted feature, and a recording segment texture weighted feature according to the mel feature, the mel-cepstral coefficient feature, the chroma feature, the recording segment texture feature, and the attention weight of each feature; The mel-weighted feature, the mel-cephalometric coefficient weighted feature, the chroma weighted feature, and the recording segment texture weighted feature are fused and calculated to generate the spectrum feature of the recording to be segmented.

4. The method according to claim 1, wherein The prior information is also related to the gender and / or age of the recording object; and determining the labeling information of each time block by a hierarchical clustering algorithm based on the distance matrix and the prior information related to the number of the recording objects includes: Determine, according to the distance matrix, two first time blocks whose distance is less than a preset distance threshold; If there is a significant conflict between the gender characteristics and / or age characteristics of the recording subjects corresponding to the two first time blocks, skipping the clustering of the two first time blocks; If there is a significant conflict between the gender characteristics and / or age characteristics of the recording subjects corresponding to the two first time blocks, clustering and merging the two first time blocks; updating the distance matrix according to the clustering merging results of the two first time blocks, and iterating the clustering process until the clustering result conforms to the prior information on the number of the recorded objects; According to the clustering result, labeling information of each of the time blocks is determined.

5. The method according to claim 1, wherein The acquiring of the non-silent time block according to the embedded vector array and the arrangement information in each time block includes: Normalizing the embedding vector array to eliminate dimensional differences in the embedding vector array at different dimensions; Preliminarily screening out, based on the activity status arrangement information in each arrangement information within each of the time blocks and the length of each of the time blocks, a second time block marked as "active" or "non-silent", wherein at least one recording object is in an active state within the second time block, and the length of the second time block is greater than or equal to a first length threshold; If there are time blocks with overlapping areas in the second time blocks, processing the time blocks with overlapping areas according to the similarity of the embedded vector arrays corresponding to the overlapping areas of the time blocks with overlapping areas, the activity states and frequency information of the time blocks with overlapping areas to generate a first processing result, wherein the processing includes merging processing and discarding processing; According to the first processing result, obtaining a third time block determined according to the corresponding second time block; Calculating the Euclidean distance between the embedding vector arrays corresponding to the third time block by using similarity analysis to generate a second processing result, where the second processing result is used to characterize the similarity between the embedding vector arrays corresponding to the third time block; If the second processing results corresponding to two adjacent third time blocks are less than or equal to the first similarity threshold, the two adjacent third time blocks are merged to generate the non-silent time block, and the length of the non-silent time block is greater than or equal to the third time block threshold.

6. The method according to claim 2, characterized in that The step of training an end-to-end recording object log model using the audio feature set, the first subset, the second subset, and the third subset to generate the recording segmentation model includes: Construct an end-to-end recording object log model, wherein the input of the end-to-end recording object log model is the audio features of the recording to be segmented, and the output of the end-to-end recording object log model includes the arrangement information and the embedding vector array of the at least two recording objects; the end-to-end recording object log model includes a first convolutional layer before the encoder, an encoder, and a second convolutional layer after the encoder; the first convolutional layer before the encoder is a downsampling layer for scale fusion of input features, the encoder is used to vector encode the input features, and the second convolutional layer is used to map the vector-encoded input features to a high-dimensional space; Training the end-to-end recording object log model for a preset number of rounds using the first subset, the second subset, and the third subset, and collecting a speech segmentation error rate DER and / or a loss function curve of each round of training; If the speech segmentation error rate DER of each round of training, and / or the change fluctuation of the loss function curve of each round of training is less than or equal to the preset fluctuation threshold, the training of the end-to-end recording object log model is terminated, and the recording segmentation model is generated according to the model parameters corresponding to the training results.

7. The method according to claim 3, characterized in that The step of processing the recording to be segmented by using a psychoacoustic model to obtain texture features of the recording segments of the recording to be segmented includes: Setting model parameters of the psychoacoustic model according to the recording to be segmented, the model parameters including bandwidth and frequency resolution of the filter; Performing spectrum analysis on the recording to be segmented by fast Fourier transform to obtain frequency domain information of the recording to be segmented, and constructing a spectrum graph corresponding to the recording to be segmented based on the frequency domain information; Acquire amplitude and phase information of each frequency component in the frequency domain information according to the spectrum diagram to generate target spectrum information; Processing the target spectrum information using an auditory filter bank that simulates the response of the basilar membrane of the human ear to generate filtered spectrum data, wherein the filters in the auditory filter bank are used to simulate the perceptual sensitivity and resolution of the human ear to different frequencies; Calculating the masking effect between the frequency components based on the filtered spectrum data to generate a masking threshold curve, wherein the masking threshold curve is used to represent the minimum sound level at which one sound can be masked by another sound at different frequencies; Calculating psychoacoustic parameters of the recording to be segmented based on the masking threshold curve, wherein the psychoacoustic parameters include a loudness parameter and a sharpness parameter; Extracting texture features of the initial recording segment of the recording to be segmented based on the filtered spectrum data, the masking threshold curve, and the psychoacoustic parameters, wherein the texture features of the initial recording segment include flatness, peak distribution, and harmonic structure of the target spectrum information; The texture features of the initial recording segment are smoothed to generate texture features of the recording segment to be segmented, wherein the smoothing process is used to reduce the influence of noise and outliers in the texture features of the initial recording segment.

8. An electronic device, characterized in that: include: A processor, a communication interface, and a memory, wherein the processor is communicatively connected to the communication interface and the memory respectively; The memory stores computer-executable instructions; The communication interface communicates and interacts with external devices; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images