Voice classification processing method and system for doctor-patient dialogue analysis

Abstract

The invention discloses a voice classification processing method and system for doctor-patient dialogue analysis. The method comprises the following steps: acquiring voice data of a doctor user; based on the sound data, extracting voiceprint features of the doctor user; dialogue voice data including a dialogue between the doctor user and the patient user is acquired; inputting the voiceprint features and the dialogue voice data into a trained voice recognition segmentation model to obtain voice segmentation result data; the voice segmentation result data comprises a plurality of voice text data which are respectively marked as the doctor user and the patient user. Visibly, real-time doctor-patient dialogue structured processing based on voiceprint and deep learning can be realized, the accuracy and availability of medical dialogue records are improved, and the risk of medical record errors caused by confusion of speakers in traditional transcription is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a voice classification processing method and system for analyzing doctor-patient dialogues. Background Technology

[0002] With the rapid growth in demand for smart healthcare and electronic medical records, medical institutions are increasingly focusing on improving the accuracy and usability of medical records through intelligent means. Among these, the automated recognition and annotation of doctor-patient dialogues has become a key technical issue. Current technologies typically collect the overall voice data of doctor-patient dialogues, record the content using general speech recognition or manual transcription methods, and rely on manual annotation to distinguish speakers in order to generate electronic medical records. However, existing solutions lack the real-time extraction of doctors' voiceprint features and the precise segmentation and annotation capabilities of deep learning models. This makes it difficult to accurately distinguish between doctors and patients in complex, multi-turn dialogues, resulting in insufficient accuracy and usability of the dialogue records. Speaker confusion can easily lead to errors in medical records and medical disputes, thus limiting the reliability of electronic medical records. Therefore, existing technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a speech classification processing method and system for doctor-patient dialogue analysis, which can realize real-time structured processing of doctor-patient dialogue based on voiceprint and deep learning, improve the accuracy and usability of medical dialogue records, and reduce the risk of medical record errors caused by confusion of speakers due to traditional transcription.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a speech classification processing method for doctor-patient dialogue analysis, the method comprising: Obtain voice data from doctor users; Based on the sound data, the voiceprint features of the doctor user are extracted; Acquire the voice data of the conversation between the doctor user and the patient user; The voiceprint features and the dialogue speech data are input into the trained speech recognition and segmentation model to obtain speech segmentation result data; the speech segmentation result data includes multiple speech text data labeled as the doctor user and the patient user, respectively.

[0005] As an optional implementation, in the first aspect of the invention, the speech recognition segmentation model is trained through the following steps: Obtain a Chinese speech dataset; the Chinese speech dataset is the AISHELL-4 Chinese speech dataset publicly available by OpenSLR; The audio data in the Chinese speech dataset is preprocessed to obtain the processed data; Based on the processed data and the basic model of the ECAPA-TDNN architecture, the speech recognition and segmentation model is obtained through iterative training.

[0006] As an optional implementation, in the first aspect of the invention, the preprocessing includes at least one of the following processing operations: Perform resampling to the preset frequency; Mono conversion; Endpoint monitoring; And extract Mel spectrum features; the parameters of the Mel spectrum features include: 80 Mel frequency bands, 512-point FFT window, 160-point frame shift and 25ms window length.

[0007] As an optional implementation, in the first aspect of the invention, the iterative training includes: Using the x-vector model with 4.5% EER as the pre-trained model, the SE-Res2Net module and attention statistical pooling were introduced. Cross-entropy loss and label smoothing were used to enhance inter-class separation. SpecAugment, MUSAN noise injection, and RIR reverberation simulation were added to enhance the data, and the training was carried out for 100 epochs.

[0008] As an optional implementation, in the first aspect of the present invention, the step of inputting the voiceprint features and the dialogue speech data into a trained speech recognition and segmentation model to obtain speech segmentation result data includes: The voiceprint features and each speech segment of the dialogue speech data are input into the trained speech recognition and segmentation model to obtain the speaker label corresponding to each of the output speech segments; All the speech segments are clustered using a spectral clustering algorithm to obtain multiple speech segment groups; The cosine similarity algorithm is used to determine whether adjacent groups of speech segments are spoken by the same person. If so, the annotation is updated to obtain the speaker annotation corresponding to each speech segment. Based on the speech-to-text model, the speech text corresponding to each speech segment is identified; Determine the corresponding speech text, speaker annotation, and timestamp annotation for each speech segment, and output the speech segmentation result data.

[0009] As an optional implementation, in the first aspect of the present invention, the speech-to-text model is trained through the following steps: Zero-shot inference was performed using the Long From ASR model, which was trained on the AISHELL-4 test set. The encoder layers are set to 24, the first 12 layers of the encoder are frozen to retain general acoustic characteristics, and only the last 12 layers of the encoder and all decoders are fine-tuned. The learning rate was set to 1e-5, the label smoothing coefficient ε=0.1, and the speech-to-text model was obtained after 10 epochs of fine-tuning training.

[0010] As an optional implementation, in the first aspect of the present invention, acquiring the doctor user's voice data includes: Based on the user parameters of the doctor users, match multiple historical doctor dialogue data with similar user parameters in the historical doctor-patient dialogue database; Based on the speech-to-text model, the historical dialogue text corresponding to each of the historical doctor dialogue data is determined. Based on the intersection calculation algorithm, the high-frequency text portion of all the historical dialogue texts is calculated to obtain the reference text; The guiding text, including the reference text, is played to the doctor user through a playback device so that the doctor user reads the reference text aloud, thereby obtaining the corresponding sound data; the voiceprint features are calculated through the following steps: Calculate the average value of the speech similarity between the sound data and each of the historical doctor dialogue data to obtain the corresponding speech similarity parameters; The sound data and the speech similarity parameters are input into the trained voiceprint extraction model to obtain the corresponding voiceprint features; the voiceprint extraction model is trained using a training dataset that includes multiple training sound data and corresponding historical speech similarity parameters and voiceprint feature annotations.

[0011] As an optional implementation, in the first aspect of the present invention, the method further includes: In response to a correction instruction for the speech segmentation data, a speaker annotation correction operation is performed on the speech segmentation data; the speaker annotation correction operation includes: Acquire the doctor user's browsing operations on the speech segmentation data; Based on the operation recognition algorithm, the time points that belong to the hesitant operation in the browsing operation are identified, and multiple hesitant time points are obtained. Obtain the browsing audio segment corresponding to each hesitation time point in the speech segmentation data; For each of the browsing audio segments, calculate the first text coherence parameter between the audio text corresponding to the browsing audio segment and the audio text of the previous audio segment; Calculate the second text coherence parameter between the speech text corresponding to the browsed speech segment and the speech text of the previous speech segment; the first text coherence parameter and the second text coherence parameter are predicted by a trained text coherence prediction model; the text coherence prediction model is trained by a training dataset including multiple training speech text pairs and corresponding coherence annotations. Calculate the average value of the first text coherence parameter and the second text coherence parameter, and determine whether the average value of the parameters is greater than a preset parameter threshold; If so, the speaker tag corresponding to the browsed audio segment is determined as the one that appears most frequently in the nearby tag set; the nearby tag set is the set of speaker tags corresponding to the browsed audio segment before and after a preset number of the browsed audio segment. If not, the speaker tag corresponding to the viewed audio segment is determined to be the one that appears least frequently in the nearby tag set.

[0012] A second aspect of this invention discloses a speech classification processing system for doctor-patient dialogue analysis, the system comprising: The acquisition module is used to acquire the voice data of doctor users; The extraction module is used to extract the voiceprint features of the doctor user based on the sound data; The acquisition module is used to acquire dialogue voice data, including the conversation between the doctor user and the patient user; The recognition module is used to input the voiceprint features and the dialogue speech data into a trained speech recognition and segmentation model to obtain speech segmentation result data; the speech segmentation result data includes multiple speech text data labeled as the doctor user and the patient user, respectively.

[0013] As an optional implementation, in a second aspect of the invention, the speech recognition segmentation model is trained through the following steps: Obtain a Chinese speech dataset; the Chinese speech dataset is the AISHELL-4 Chinese speech dataset publicly available by OpenSLR; The audio data in the Chinese speech dataset is preprocessed to obtain the processed data; Based on the processed data and the basic model of the ECAPA-TDNN architecture, the speech recognition and segmentation model is obtained through iterative training.

[0014] As an optional implementation, in a second aspect of the invention, the preprocessing includes at least one of the following processing operations: Perform resampling to the preset frequency; Mono conversion; Endpoint monitoring; And extract Mel spectrum features; the parameters of the Mel spectrum features include: 80 Mel frequency bands, 512-point FFT window, 160-point frame shift and 25ms window length.

[0015] As an optional implementation, in a second aspect of the invention, the iterative training includes: Using the x-vector model with 4.5% EER as the pre-trained model, the SE-Res2Net module and attention statistical pooling were introduced. Cross-entropy loss and label smoothing were used to enhance inter-class separation. SpecAugment, MUSAN noise injection, and RIR reverberation simulation were added to enhance the data, and the training was carried out for 100 epochs.

[0016] As an optional implementation, in a second aspect of the invention, the specific method by which the recognition module inputs the voiceprint features and the dialogue speech data into a trained speech recognition and segmentation model to obtain speech segmentation result data includes: The voiceprint features and each speech segment of the dialogue speech data are input into the trained speech recognition and segmentation model to obtain the speaker label corresponding to each of the output speech segments; All the speech segments are clustered using a spectral clustering algorithm to obtain multiple speech segment groups; The cosine similarity algorithm is used to determine whether adjacent groups of speech segments are spoken by the same person. If so, the annotation is updated to obtain the speaker annotation corresponding to each speech segment. Based on the speech-to-text model, the speech text corresponding to each speech segment is identified; Determine the corresponding speech text, speaker annotation, and timestamp annotation for each speech segment, and output the speech segmentation result data.

[0017] As an optional implementation, in a second aspect of the invention, the speech-to-text model is trained through the following steps: Zero-shot inference was performed using the Long From ASR model, which was trained on the AISHELL-4 test set. The encoder layers are set to 24, the first 12 layers of the encoder are frozen to retain general acoustic characteristics, and only the last 12 layers of the encoder and all decoders are fine-tuned. The learning rate was set to 1e-5, the label smoothing coefficient ε=0.1, and the speech-to-text model was obtained after 10 epochs of fine-tuning training.

[0018] As an optional implementation, in the second aspect of the present invention, the specific method by which the acquisition module acquires the doctor user's voice data includes: Based on the user parameters of the doctor users, match multiple historical doctor dialogue data with similar user parameters in the historical doctor-patient dialogue database; Based on the speech-to-text model, the historical dialogue text corresponding to each of the historical doctor dialogue data is determined. Based on the intersection calculation algorithm, the high-frequency text portion of all the historical dialogue texts is calculated to obtain the reference text; The guiding text, including the reference text, is played to the doctor user through a playback device so that the doctor user reads the reference text aloud, thereby obtaining the corresponding sound data; the voiceprint features are calculated through the following steps: Calculate the average value of the speech similarity between the sound data and each of the historical doctor dialogue data to obtain the corresponding speech similarity parameters; The sound data and the speech similarity parameters are input into the trained voiceprint extraction model to obtain the corresponding voiceprint features; the voiceprint extraction model is trained using a training dataset that includes multiple training sound data and corresponding historical speech similarity parameters and voiceprint feature annotations.

[0019] As an optional implementation, in a second aspect of the invention, the system is further configured to perform the following steps: In response to a correction instruction for the speech segmentation data, a speaker annotation correction operation is performed on the speech segmentation data; the speaker annotation correction operation includes: Acquire the doctor user's browsing operations on the speech segmentation data; Based on the operation recognition algorithm, the time points that belong to the hesitant operation in the browsing operation are identified, and multiple hesitant time points are obtained. Obtain the browsing audio segment corresponding to each hesitation time point in the speech segmentation data; For each of the browsing audio segments, calculate the first text coherence parameter between the audio text corresponding to the browsing audio segment and the audio text of the previous audio segment; Calculate the second text coherence parameter between the speech text corresponding to the browsed speech segment and the speech text of the previous speech segment; the first text coherence parameter and the second text coherence parameter are predicted by a trained text coherence prediction model; the text coherence prediction model is trained by a training dataset including multiple training speech text pairs and corresponding coherence annotations. Calculate the average value of the first text coherence parameter and the second text coherence parameter, and determine whether the average value of the parameters is greater than a preset parameter threshold; If so, the speaker tag corresponding to the browsed audio segment is determined as the one that appears most frequently in the nearby tag set; the nearby tag set is the set of speaker tags corresponding to the browsed audio segment before and after a preset number of the browsed audio segment. If not, the speaker tag corresponding to the viewed audio segment is determined to be the one that appears least frequently in the nearby tag set.

[0020] A third aspect of this invention discloses another speech classification and processing system for doctor-patient dialogue analysis, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the speech classification processing method for doctor-patient dialogue analysis disclosed in the first aspect of the present invention.

[0021] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the speech classification processing method for doctor-patient dialogue analysis disclosed in the first aspect of the present invention.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention extracts voiceprint features from doctor user voice data and combines them with dialogue voice data input into a trained speech recognition and segmentation model to achieve accurate segmentation and speaker labeling of doctor-patient dialogues. This enables real-time structured processing of doctor-patient dialogues based on voiceprints and deep learning, improving the accuracy and usability of medical dialogue records and reducing the risk of errors in medical records caused by confusion of speakers in traditional transcription. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a speech classification processing method for doctor-patient dialogue analysis disclosed in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of a voice classification and processing system for doctor-patient dialogue analysis disclosed in an embodiment of the present invention.

[0026] Figure 3This is a schematic diagram of another speech classification and processing system for doctor-patient dialogue analysis disclosed in an embodiment of the present invention.

[0027] Figure 4 This is a flowchart illustrating an algorithm for segmenting doctor-patient dialogue disclosed in an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] This invention discloses a speech classification and processing method and system for doctor-patient dialogue analysis. By acquiring doctor user voice data to extract voiceprint features and combining this with dialogue speech data input into a trained speech recognition and segmentation model, accurate segmentation and speaker labeling of doctor-patient dialogue are achieved. This enables real-time structured processing of doctor-patient dialogue based on voiceprints and deep learning, improving the accuracy and usability of medical dialogue records and reducing the risk of errors in medical records caused by speaker confusion in traditional transcription methods. Detailed explanations follow.

[0032] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a speech classification processing method for doctor-patient dialogue analysis disclosed in an embodiment of the present invention. Wherein, Figure 1The described voice classification processing method for doctor-patient dialogue analysis can be applied to data processing systems / data processing devices / data processing servers (including local processing servers or cloud processing servers). Figure 1 As shown, the speech classification processing method for doctor-patient dialogue analysis may include the following operations: 101. Obtain the voice data of doctor users.

[0033] Optionally, the audio data can be a 10-second greeting recorded when the doctor logs into the system, a clip of a historical medical record, or a clean voice captured by a real-time microphone; this invention does not impose any limitations.

[0034] 102. Extract voiceprint features from doctor users based on voice data.

[0035] Optionally, the voiceprint feature can be a 512-dimensional or 256-dimensional x-vector, d-vector, or ECAPA-TDNN embedding vector, and this invention does not impose any limitations.

[0036] 103. Obtain dialogue voice data, including conversations between doctor users and patient users.

[0037] Optionally, the dialogue voice data can be in mono 16kHz WAV format, stereo conference recording, or real-time audio stream from a remote consultation platform; this invention does not limit the scope of the data.

[0038] 104. Input the voiceprint features and dialogue speech data into the trained speech recognition and segmentation model to obtain speech segmentation result data.

[0039] Optionally, the speech segmentation results data include multiple speech-text data points labeled as doctor users and patient users, respectively.

[0040] Optionally, the speech recognition segmentation model can be an end-to-end speaker segmentation system of ECAPA-TDNN+Conformer, which outputs speaker labels every 200ms. This invention does not limit this.

[0041] As can be seen, the above-mentioned embodiments of the invention extract voiceprint features by acquiring doctor user voice data and combining them with dialogue voice data input into a trained speech recognition and segmentation model to achieve accurate segmentation and speaker labeling of doctor-patient dialogue. This enables real-time structured processing of doctor-patient dialogue based on voiceprint and deep learning, improves the accuracy and usability of medical dialogue records, and reduces the risk of errors in medical records caused by confusion of speakers in traditional transcription.

[0042] As an optional embodiment, the speech recognition segmentation model in the above steps is trained through the following steps: Obtain a Chinese speech dataset; Optionally, the Chinese speech dataset is the AISHELL-4 Chinese speech dataset publicly available by OpenSLR; The audio data in the Chinese speech dataset is preprocessed to obtain the processed data. Based on the processed data and the basic model of the ECAPA-TDNN architecture, a speech recognition and segmentation model is obtained through iterative training.

[0043] Optionally, the dataset may include multi-speaker conference scenarios with a total duration of over 200 hours, but this invention does not limit the scope of the dataset.

[0044] Optionally, preprocessing includes at least one of the following processing operations: Perform resampling to the preset frequency; Mono conversion; Endpoint monitoring; And extract Mel spectral features; the parameters of the Mel spectral features include: 80 Mel bands, 512-point FFT window, 160-point frame shift and 25ms window length.

[0045] Optional, iterative training includes: Using the x-vector model with 4.5% EER as the pre-trained model, the SE-Res2Net module and attention statistical pooling were introduced. Cross-entropy loss and label smoothing were used to enhance inter-class separation. SpecAugment, MUSAN noise injection, and RIR reverberation simulation were added to enhance the data, and the training was carried out for 100 epochs.

[0046] As can be seen, through the above optional embodiments, a highly robust speech recognition and segmentation model is obtained by iterative training based on the publicly available AISHELL-4 dataset and using the ECAPA-TDNN basic model. This solution achieves high-precision segmentation of complex overlapping speech and noisy environments in Chinese medical scenarios on the basis of accurate speaker segmentation, reducing the risk of segmentation failure due to insufficient model generalization ability.

[0047] As an optional embodiment, the step described above, inputting the voiceprint features and dialogue speech data into the trained speech recognition and segmentation model to obtain speech segmentation result data, includes: Input each speech segment of the voiceprint features and dialogue speech data into the trained speech recognition and segmentation model to obtain the speaker label corresponding to each speech segment in the output; All speech segments are clustered using a spectral clustering algorithm to obtain multiple speech segment groups. The cosine similarity algorithm is used to determine whether adjacent speech segments are spoken by the same person. If so, the annotation is updated to obtain the speaker annotation for each speech segment. Based on the speech-to-text model, the speech text corresponding to each speech segment is identified; Determine the corresponding speech text, speaker annotation, and timestamp annotation for each speech segment, and output the speech segmentation result data.

[0048] Optionally, the speech-to-text model is trained through the following steps: Zero-shot inference was performed using the Long From ASR model, which was trained on the AISHELL-4 test set. The encoder layers are set to 24, the first 12 layers of the encoder are frozen to retain general acoustic characteristics, and only the last 12 layers of the encoder and all decoders are fine-tuned. The learning rate was set to 1e-5, the label smoothing coefficient ε=0.1, and the speech-to-text model was obtained after 10 epochs of fine-tuning training.

[0049] As can be seen, through the above optional embodiments, by combining voiceprint features to perform coarse segmentation of the model first, then using spectral clustering + cosine similarity to merge and correct adjacent segments, and finally combining with a speech-to-text model to complete high-quality speech transcription and timestamp annotation, this solution significantly improves the integrity and accuracy of structured data of doctor-patient dialogues through multi-stage fusion algorithms on the basis of accurate speech segmentation output, and reduces the risk of annotation errors caused by the limitations of a single model.

[0050] In one specific implementation scheme, an algorithm for segmenting doctor-patient dialogue is implemented based on the above technical solution. Its flowchart can be found in [reference needed]. Figure 4 The steps include: Step 1: Record the doctor's voiceprint Recording the doctor's voiceprint (voiceprint registration phase): When using the system for the first time, the doctor enters the voiceprint recording interface by clicking the "Doctor's Voiceprint" button in the upper right corner of the interface. After the system pops up the "Record Doctor's Voiceprint" window, the doctor speaks alone for about 5-10 seconds, such as reading an opening greeting or self-introduction. During this process, the device automatically collects audio signals and extracts voiceprint feature parameters. After recording is complete, the doctor clicks to stop recording. The system saves this voiceprint data as the doctor's personal voiceprint template for speaker recognition and role differentiation in subsequent conversations. This step ensures that the system can accurately distinguish the voice source of the doctor and the patient, providing basic support for subsequent real-time speech recognition and dialogue annotation, and achieving automatic identification and high-precision separation of doctor and patient roles.

[0051] Based on a speech stream speaker segmentation and classification model for two-person dialogue, voiceprint information is obtained through a recording device. The audio data preprocessing process includes: The audio from the AISHELL-4 Chinese speech dataset publicly available by OpenSLR was resampled to 16kHz, mono-channel conversion was performed, endpoint detection was performed, and Mel-spectrogram features were extracted. The parameters of the Mel-spectrogram features included: 80 Mel bands, 512-point FFT window, 160-point frame shift, and 25ms window length. The preprocessed data was then used to train the speech stream speaker segmentation and classification model for the two-person dialogue.

[0052] Optionally, the computer used for model training and the computer used as the computing hub are the same computer. The only difference is that the system used for training is WSL-Ubuntu-22.04, while the system used as the computing hub is Windows 11-version 23H2. The Python environment used is created using Anaconda3, and the CUDA version is 12.2. The detailed computer configuration is as follows: CPU - AMD Ryzen 9 7950x, memory - DDR5 6000Mhz 32G, GPU - NVIDIA GeForce RTX 4090.

[0053] Step 2: Create a session The doctor clicks the "New Conversation" button in the lower left corner of the interface. The system immediately generates a conversation record with a unique number and switches the current status to "Available for Recording." The interface then displays that it has entered the recording state, ready to initiate the voice acquisition and real-time recognition functions. This step aims to create an independent data folder for each doctor-patient conversation, ensuring the uniqueness and traceability of the conversation, and providing a data carrier and management foundation for subsequent recording, recognition, transcription, and export operations. Step 3: Start recording and real-time recognition After the doctor clicks "Start Recording," the system activates the voice acquisition module to capture the audio stream of the entire doctor-patient conversation in real time. Simultaneously, the voiceprint recognition module automatically identifies the speaker, distinguishing between "doctor" and "patient," and displays the corresponding waveform and role label on the interface in real time. The system's voice recognition function synchronously transcribes both parties' voices into text, displaying it in real-time in the interface's text area, achieving instant transcription and role labeling of the conversation.

[0054] The speech stream speaker segmentation and classification model based on two-person dialogue, after preprocessing the audio data, includes the following training steps: The speech transcription model (Long From ASR) of iFlytek was applied, and the training set used was the AISHELL-4 Chinese speech dataset publicly available by OpenSLR, to obtain the aforementioned two-person dialogue speech separation model.

[0055] After 50 epochs of training, the equal error rate (EER) for speaker verification remained stable at 4.5% for an extended period. Further analysis of the training data revealed problems not only with the EER but also with the accuracy of speaker switching detection in real-world two-person dialogue scenarios. The system's recognition rate for rapid speaker switching (<1 second) segments was only 68.3%, with many short-duration speeches being incorrectly assigned. Visualization of the embedding space (t-SNE dimensionality reduction) revealed significant overlap in cluster boundaries between different speakers, particularly between speakers of the same gender with similar voices (such as two female interviewers), where the intra-class distance was excessively large, reaching 0.58, while the ideal value should be less than 0.3. Further investigation revealed that the main issues stemmed from: insufficient capacity of the x-vector model (only 7 layers of TDNN), limited receptive field, inability to fully model long-term speech features, a simplistic training strategy, and the lack of explicit inter-class separation constraints in traditional Softmax. Therefore, in later training phases, it was decided to switch to the ECAPA-TDNN architecture and re-annotate speaker segments from the AISHELL-4 dataset. The x-vector model with a 4.5% EER was used as the pre-trained model for subsequent training. The SE-Res2Net module and attention statistical pooling were introduced, and cross-entropy loss with label smoothing (smoothing=0.1) was employed to enhance inter-class separation. Six enhancement methods, including SpecAugment, MUSAN noise injection, and RIR reverberation simulation, were added to augment the data. After 100 epochs of training, the model's EER on the validation set steadily decreased to 2.8%, a 37.8% improvement compared to the initial x-vector model. Speaker switching detection accuracy increased to 92.3%, and short-term speech recognition accuracy improved from 68.3% to 89.8%.

[0056] Then, speaker clustering is performed. The extracted embedding vectors are grouped using the Spectral Clustering algorithm, and the speaker label (Speaker A / Speaker B) for each audio segment in the received data is returned. The system uses cosine similarity to determine whether adjacent segments belong to the same speaker. When the similarity is lower than a threshold, a speaker switching event is triggered, and the timestamp label is updated.

[0057] The speech stream speaker segmentation and classification model based on two-person dialogue is processed by the speech collection device; the training steps of the speech transcription module model include: Using iFlytek's Long From ASR speech transcription model, with a learning rate of 1e-5, and after 10 epochs of training, the model loss reached 0.2319, resulting in the speech stream speaker segmentation and classification model for the two-person dialogue.

[0058] Specifically: The speech-to-text module initially used a Long From ASR model for zero-shot inference. On the AISHELL-4 test set, the base error rate (CER) without fine-tuning was 8.9%, which is better than the traditional HMM-DNN system (CER≈12%), but the error rate for specialized terminology and colloquial expressions was relatively high. Therefore, in subsequent training, the number of encoder layers was increased from 6 to 24, the first 12 layers of the encoder were frozen (preserving general acoustic features), and only the last 12 layers of the encoder and all decoders were fine-tuned. The learning rate was set to 1e-5, and the label smoothing coefficient ε=0.1. After 10 epochs of fine-tuning (approximately 15 hours), the CER on the validation set decreased from the initial 7.8% to 6.1%, a relative improvement of 21.8%. The accuracy of specialized terminology recognition increased from 78% to 91%, and the error rate of colloquial expressions decreased from 15% to 9%. The technical requirements were met.

[0059] Step 4: Stop and save the session When the doctor clicks the "Stop Recording" button, the system will end the current recording stream and automatically and completely save all data from this session, including audio content, transcribed text, speaker markers, timestamps, and other information. This data will be packaged and stored in a system-designated directory. After successful saving, the session will automatically be added to the "History" list, where the doctor can view the recorded content, play back the recording, or delete the session.

[0060] Step 5: File Export and Sharing In the history session list, doctors can select any record and click "Export Session". The system will automatically generate files in various formats, such as text (.txt), tables (.csv), and audio (.wav). If a USB flash drive is inserted, the system will automatically recognize it and write the exported file to external storage; if no USB flash drive is inserted, the file will be saved to the device's local directory. If the device is connected to the internet, it can also generate an access address through the local area network sharing function, allowing other terminals to browse or download related files.

[0061] Furthermore, a device for implementing the aforementioned algorithm was also implemented, employing a modular system architecture design. It consists of functional modules such as a main control processing module, an audio acquisition module, a display and interaction module, and a storage and export module. These modules work collaboratively through standardized interfaces to achieve a complete functional chain encompassing voice acquisition, real-time transcription, data storage, interactive control, and data export. Specifically, it includes: 1. Main control processing module: Core components: 64-bit quad-core SoC processor, 8GB RAM, running Linux operating system. Main functions: It runs a speech recognition engine in real time, converting audio streams into text data, and supports both offline and online cloud recognition modes.

[0062] The system executes a voiceprint recognition algorithm to distinguish between the speech of doctors and patients, and automatically labels the speaker.

[0063] Manage the device's data storage, query, retrieval, and export functions, and maintain the session record database.

[0064] Control the display UI rendering and touch interaction response to provide a user-friendly interface.

[0065] It provides HTTP file sharing services, supporting devices within the local area network to access and download session data.

[0066] 2. Audio Acquisition Module Core components: Dual-microphone omnidirectional MEMS array + hardware AEC noise reduction circuit + AGC automatic gain control Main functions: High-fidelity sound pickup over long distances, with an effective pickup distance of 5-8 meters, suitable for the indoor environment of outpatient clinics.

[0067] MEMS microphone array technology is used to achieve omnidirectional sound pickup and beamforming, thereby enhancing the target human voice signal.

[0068] Integrated hardware AEC echo cancellation circuitry suppresses speaker echo interference (such as the prompts played by the device itself).

[0069] By using automatic gain control (AGC) and noise suppression algorithms, the impact of environmental noise is reduced (noise suppression ≥20dB), thereby improving the quality of the speech signal.

[0070] It supports 48 kHz / 16-bit high-quality audio sampling to ensure accurate speech recognition.

[0071] 3. Display Interaction Module Core component: 1024×600 resolution IPS capacitive touchscreen Main functions: It displays speech-to-text in real time with a transcription latency of less than 500ms and provides instant feedback.

[0072] It provides a touch interface and supports functions such as session start, pause, end, query, delete, and export.

[0073] Display device status information, including network connection status, storage capacity, voiceprint registration status, etc.

[0074] It supports multi-touch operation, enhancing the user interaction experience.

[0075] 4. Storage and Export Module Core components: Internal eMMC / SSD storage + USB Type-C data interface Main functions: Transcribed text (JSON format) and original audio files (WAV format) are stored locally to ensure data integrity and traceability.

[0076] Data is organized and stored according to dimensions such as session ID, date, and patient ID, supporting fast retrieval and management.

[0077] It supports direct connection to a USB flash drive via USB Type-C interface for file export without the need to install drivers.

[0078] Provides web file sharing services, allowing devices on the local area network to access, preview, and download session data via a browser.

[0079] 5. Network communication module Core component: Dual-band Wi-Fi module (2.4GHz / 5.8GHz) Main functions: It supports internet connectivity and utilizes a cloud-based speech recognition engine to improve recognition accuracy and response speed.

[0080] It supports local area network communication and provides web file sharing services and remote management interfaces.

[0081] Supports OTA (Over-The-Air) firmware upgrades, facilitating feature iteration and bug fixing.

[0082] 6. Power Management Module Core component: USB Type-C power supply interface (5V / 3A) Main functions: It provides a stable 5V power supply, supporting the device to work continuously for a long time.

[0083] Low power consumption design, operating power consumption is about 1W (5V / 0.2A), and standby power consumption is <0.1W.

[0084] In summary, the above-described implementation plan achieves a complete integrated workflow, combining "conversation creation, recording capture, real-time recognition, character segmentation, data storage, and export sharing" into a single system. Doctors no longer need to switch between multiple software programs, significantly reducing operational complexity and improving clinical efficiency. The system features real-time voiceprint character recognition, automatically distinguishing between "doctor" and "patient" identities during recording, avoiding the character confusion issues common in traditional voice transcription, and making case processing and data analysis more accurate. Through real-time speech transcription and visualization, the system, along with waveform graphs and character tags, enhances operational transparency and interactive experience. In terms of data management, the system automatically generates a unique number for each conversation and saves complete data packages including audio, text, timestamps, and character information, ensuring complete records, consistent structure, and easy retrieval and backtracking. Furthermore, the system supports local storage, USB drive export, and LAN sharing, meeting the needs of various scenarios such as departmental communication, remote consultation, and teaching demonstrations. Therefore, this solution, by integrating real-time recognition, character segmentation, and automatic data management technologies, significantly improves the efficiency, accuracy, and usability of doctor-patient dialogue recordings, demonstrating outstanding application value and promotional significance.

[0085] As an optional embodiment, the step of obtaining the doctor user's voice data in the above steps includes: Based on the user parameters of the doctor users, match multiple historical doctor dialogue data with similar user parameters in the historical doctor-patient dialogue database; Based on the speech-to-text model, the historical dialogue text corresponding to each historical doctor dialogue data is determined. Based on the intersection calculation algorithm, the high-frequency text portion of all historical dialogue texts is calculated to obtain the reference text; The guiding text, which includes reference text, is played to the doctor user through a playback device so that the doctor user can read the reference text aloud and obtain the corresponding sound data.

[0086] Optionally, the user parameters may include hospital level, department, professional title, gender, age group, and accent region; however, this invention does not impose any limitations on these parameters.

[0087] Optionally, multiple historical doctor dialogues with similar user parameters can be filtered based on parameter similarity and similarity thresholds.

[0088] Optionally, the high-frequency text portion may be an opening remark or a fixed questioning phrase that appears ≥60% of the time, such as "Hello, where do you feel unwell?" This invention does not impose any limitations on this.

[0089] Optionally, voiceprint features are calculated through the following steps: Calculate the average speech similarity between the audio data and each historical doctor's dialogue data to obtain the corresponding speech similarity parameters; The sound data and speech similarity parameters are input into the trained voiceprint extraction model to obtain the corresponding voiceprint features.

[0090] Optionally, the voiceprint extraction model is trained using a training dataset that includes multiple training voice data and corresponding historical speech similarity parameters and voiceprint feature annotations.

[0091] Optionally, the voiceprint extraction model can be a 512-dimensional embedding model trained with ResNet34-SE+Angular Prototypical loss, trained on a database of 100,000 doctors' voices. This invention does not impose any limitations.

[0092] As can be seen, through the above optional embodiments, by matching similar doctors through the historical doctor-patient dialogue database and extracting high-frequency text to guide doctors to read aloud, and combining the voice similarity parameter weighted input voiceprint extraction model to obtain more accurate doctor voiceprint features, this solution significantly improves the voiceprint recognition accuracy when used for the first time through the adaptive voiceprint registration process of "historical similarity + guided reading" on the basis of accurate voiceprint extraction, and reduces the risk of subsequent segmentation errors caused by inaccurate voiceprints during cold start.

[0093] As an optional embodiment, the method further includes the following steps: In response to a correction command for the speech segmentation data, a speaker annotation correction operation is performed on the speech segmentation data; the speaker annotation correction operation includes: Acquire doctor users' browsing operations on speech segmentation data; Based on the operation recognition algorithm, the time points that belong to the hesitant operation in the browsing operation are identified, and multiple hesitant time points are obtained. Obtain the browsing audio segment corresponding to each hesitation time point in the speech segmentation data; For each audio segment being viewed, calculate the first text coherence parameter between the audio text corresponding to the audio segment being viewed and the audio text of the previous audio segment. Calculate the second text coherence parameter between the audio text corresponding to the current audio segment and the audio text of the previous audio segment.

[0094] Calculate the average value of the first text coherence parameter and the second text coherence parameter, and determine whether the average value of the parameters is greater than a preset parameter threshold. If so, the speaker annotation corresponding to the browsed audio segment is determined as the one that appears most frequently in the nearby annotation set; optionally, the nearby annotation set is the set of speaker annotations corresponding to a preset number of browsed audio segments before and after the browsed audio segment. If not, the speaker tag corresponding to the viewed audio segment is determined to be the one that appears least frequently in the nearby tag set.

[0095] Optionally, the first text coherence parameter and the second text coherence parameter are predicted by a trained text coherence prediction model.

[0096] Optionally, the text coherence prediction model is trained using a training dataset that includes multiple training speech-text pairs and their corresponding coherence annotations.

[0097] Optionally, the text coherence prediction model can be RoBERTa-base Chinese version + linear classification head, fine-tuned on the doctor-patient dialogue coherence dataset, and output a coherence probability of 0-1. This invention does not limit this.

[0098] Optionally, a hesitation operation can be an operation that involves hovering the mouse for more than 3 seconds or repeatedly dragging the cursor.

[0099] To illustrate the above solution, consider the following specific implementation case: On March 20, 2025, Dr. Li from the cardiology department used the system for a remote consultation. Upon login, the system played a guiding message: "Hello, where do you feel unwell? What symptoms have you had recently?" After Dr. Li read it aloud, the system extracted his 512-dimensional voiceprint features.

[0100] During the consultation, the system collected real-time audio recordings of the conversation between Dr. Li and patient Wang. The speech recognition and segmentation model outputs a speaker label and transcribed text every 1.5 seconds, with an initial accuracy of 96.3%. However, a patient's family member interrupted between seconds 127 and 132, causing three segments to be misidentified as Dr. Li's speech.

[0101] During the playback, Dr. Li hovered the mouse over the 127-132 second range for 8 seconds and dragged it back and forth, which the system detected as a hesitant action. The system automatically extracted the text "Yes, yes, I sometimes experience chest tightness" from this segment and calculated its coherence with the preceding and following text. The continuity with the preceding section (Dr. Li's speech) is 0.23; The continuity with the latter part (the patient's speech) is 0.91; The average value was 0.57 < 0.6, so the system judged it as a misjudgment and automatically corrected the label of the segment to "Patient". At the same time, it also corrected the similar misjudgments of the two segments before and after it.

[0102] Ultimately, the entire dialogue was 100% correctly segmented into Doctor / Patient, generating a structured consultation record with perfect speaker labels, achieving 100% accuracy and greatly reducing the burden of manual annotation.

[0103] As can be seen, through the above optional embodiments, by capturing the doctor's hesitant behavior when browsing the segmentation results, and combining the prediction model of the coherence between the preceding and following texts to intelligently judge and automatically correct potential annotation errors, this solution achieves a semi-automatic error correction mechanism of human-machine collaboration on the basis of accurate doctor-patient dialogue segmentation. This significantly reduces the workload of manual full proofreading while improving the final annotation accuracy rate, and reduces the risk of incorrect medical record attribution due to doctor's negligence or model misjudgment.

[0104] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of a speech classification and processing system for doctor-patient dialogue analysis disclosed in an embodiment of the present invention. Figure 2 The described voice classification and processing system for doctor-patient dialogue analysis can be applied to data processing systems / data processing devices / data processing servers (wherein, the server includes local processing servers or cloud processing servers). For example... Figure 2 As shown, the speech classification and processing system for doctor-patient dialogue analysis may include: Module 201 is used to acquire the voice data of doctor users.

[0105] Extraction module 202 is used to extract the voiceprint features of doctor users based on sound data.

[0106] The acquisition module 203 is used to acquire dialogue voice data, including the conversation between the doctor user and the patient user.

[0107] The recognition module 204 is used to input the voiceprint features and dialogue speech data into the trained speech recognition and segmentation model to obtain speech segmentation result data.

[0108] Optionally, the speech segmentation results data include multiple speech-text data points labeled as doctor users and patient users, respectively.

[0109] As can be seen, the above-mentioned embodiments of the invention extract voiceprint features by acquiring doctor user voice data and combining them with dialogue voice data input into a trained speech recognition and segmentation model to achieve accurate segmentation and speaker labeling of doctor-patient dialogue. This enables real-time structured processing of doctor-patient dialogue based on voiceprint and deep learning, improves the accuracy and usability of medical dialogue records, and reduces the risk of errors in medical records caused by confusion of speakers in traditional transcription.

[0110] As an optional implementation, the speech recognition segmentation model is trained through the following steps: Obtain a Chinese speech dataset; optionally, the Chinese speech dataset is the AISHELL-4 Chinese speech dataset publicly available by OpenSLR. The audio data in the Chinese speech dataset is preprocessed to obtain the processed data. Based on the processed data and the basic model of the ECAPA-TDNN architecture, a speech recognition and segmentation model is obtained through iterative training.

[0111] As can be seen, through the above optional embodiments, a highly robust speech recognition and segmentation model is obtained by iterative training based on the publicly available AISHELL-4 dataset and using the ECAPA-TDNN basic model. This solution achieves high-precision segmentation of complex overlapping speech and noisy environments in Chinese medical scenarios on the basis of accurate speaker segmentation, reducing the risk of segmentation failure due to insufficient model generalization ability.

[0112] As an optional embodiment, preprocessing includes at least one of the following processing operations: Perform resampling to the preset frequency; Mono conversion; Endpoint monitoring; And extract Mel spectral features; the parameters of the Mel spectral features include: 80 Mel bands, 512-point FFT window, 160-point frame shift and 25ms window length.

[0113] As can be seen, the above optional embodiments define the audio preprocessing steps to obtain more standardized audio data and audio features, which helps to realize the real-time structured processing of doctor-patient dialogue based on voiceprint and deep learning, and improves the accuracy and usability of medical dialogue records.

[0114] As an optional implementation, iterative training includes: Using the x-vector model with 4.5% EER as the pre-trained model, the SE-Res2Net module and attention statistical pooling were introduced. Cross-entropy loss and label smoothing were used to enhance inter-class separation. SpecAugment, MUSAN noise injection, and RIR reverberation simulation were added to enhance the data, and the training was carried out for 100 epochs.

[0115] As can be seen, by defining the details of iterative training through the above optional embodiments, it is possible to introduce the SE-Res2Net module + attention statistical pooling architecture, and combine multiple data augmentations such as SpecAugment to carry out 100 epochs of reinforcement training to obtain a highly robust speech recognition and segmentation model. This model can help realize the real-time structured processing of doctor-patient dialogue based on voiceprint and deep learning, and improve the accuracy and usability of medical dialogue records.

[0116] As an optional embodiment, the specific method by which the recognition module inputs voiceprint features and dialogue speech data into the trained speech recognition and segmentation model to obtain speech segmentation result data includes: Input each speech segment of the voiceprint features and dialogue speech data into the trained speech recognition and segmentation model to obtain the speaker label corresponding to each speech segment in the output; All speech segments are clustered using a spectral clustering algorithm to obtain multiple speech segment groups. The cosine similarity algorithm is used to determine whether adjacent speech segments are spoken by the same person. If so, the annotation is updated to obtain the speaker annotation for each speech segment. Based on the speech-to-text model, the speech text corresponding to each speech segment is identified; Determine the corresponding speech text, speaker annotation, and timestamp annotation for each speech segment, and output the speech segmentation result data.

[0117] As can be seen, through the above optional embodiments, by combining voiceprint features to perform coarse segmentation of the model first, then using spectral clustering + cosine similarity to merge and correct adjacent segments, and finally combining with a speech-to-text model to complete high-quality speech transcription and timestamp annotation, this solution significantly improves the integrity and accuracy of structured data of doctor-patient dialogues through multi-stage fusion algorithms on the basis of accurate speech segmentation output, and reduces the risk of annotation errors caused by the limitations of a single model.

[0118] As an optional implementation, the speech-to-text model is trained through the following steps: Zero-shot inference was performed using the Long From ASR model, which was trained on the AISHELL-4 test set. The encoder layers are set to 24, the first 12 layers of the encoder are frozen to retain general acoustic characteristics, and only the last 12 layers of the encoder and all decoders are fine-tuned. The learning rate was set to 1e-5, the label smoothing coefficient ε=0.1, and the speech-to-text model was obtained after 10 epochs of fine-tuning training.

[0119] As can be seen, through the above optional embodiments, the Long From ASR model strategy of zero samples + fine-tuning the last 12 layers can achieve excellent transcription performance in medical dialogue scenarios in just 10 epochs. This solution achieves low-cost and high-efficiency Chinese medical speech transcription model adaptation on the basis of accurate speech transcription, and reduces the risk of excessive computing power and data requirements caused by full parameter fine-tuning.

[0120] As an optional embodiment, the specific method by which the acquisition module acquires the doctor user's voice data includes: Based on the user parameters of the doctor users, match multiple historical doctor dialogue data with similar user parameters in the historical doctor-patient dialogue database; Based on the speech-to-text model, the historical dialogue text corresponding to each historical doctor dialogue data is determined. Based on the intersection calculation algorithm, the high-frequency text portion of all historical dialogue texts is calculated to obtain the reference text; The guiding text, including the reference text, is played to the doctor user through a playback device so that the doctor user can read the reference text aloud, thereby obtaining the corresponding voice data; the voiceprint features are calculated through the following steps: Calculate the average speech similarity between the audio data and each historical doctor's dialogue data to obtain the corresponding speech similarity parameters; The sound data and speech similarity parameters are input into the trained voiceprint extraction model to obtain the corresponding voiceprint features. The voiceprint extraction model is trained using a training dataset that includes multiple training sound data and corresponding historical speech similarity parameters and voiceprint feature annotations.

[0121] As can be seen, through the above optional embodiments, by matching similar doctors through the historical doctor-patient dialogue database and extracting high-frequency text to guide doctors to read aloud, and combining the voice similarity parameter weighted input voiceprint extraction model to obtain more accurate doctor voiceprint features, this solution significantly improves the voiceprint recognition accuracy when used for the first time through the adaptive voiceprint registration process of "historical similarity + guided reading" on the basis of accurate voiceprint extraction, and reduces the risk of subsequent segmentation errors caused by inaccurate voiceprints during cold start.

[0122] As an optional embodiment, the system is also used to perform the following steps: In response to a correction command for the speech segmentation data, a speaker annotation correction operation is performed on the speech segmentation data; the speaker annotation correction operation includes: Acquire doctor users' browsing operations on speech segmentation data; Based on the operation recognition algorithm, the time points that belong to the hesitant operation in the browsing operation are identified, and multiple hesitant time points are obtained. Obtain the browsing audio segment corresponding to each hesitation time point in the speech segmentation data; For each audio segment being viewed, calculate the first text coherence parameter between the audio text corresponding to the audio segment being viewed and the audio text of the previous audio segment. Calculate the second text coherence parameter between the speech text corresponding to the browsed speech segment and the speech text of the previous speech segment; optionally, the first text coherence parameter and the second text coherence parameter are predicted by a trained text coherence prediction model; the text coherence prediction model is trained by a training dataset including multiple training speech text pairs and corresponding coherence annotations. Calculate the average value of the first text coherence parameter and the second text coherence parameter, and determine whether the average value of the parameters is greater than a preset parameter threshold. If so, the speaker annotation corresponding to the browsed audio segment is determined as the one that appears most frequently in the nearby annotation set; optionally, the nearby annotation set is the set of speaker annotations corresponding to a preset number of browsed audio segments before and after the browsed audio segment. If not, the speaker tag corresponding to the viewed audio segment is determined to be the one that appears least frequently in the nearby tag set.

[0123] As can be seen, through the above optional embodiments, by capturing the doctor's hesitant behavior when browsing the segmentation results, and combining the prediction model of the coherence between the preceding and following texts to intelligently judge and automatically correct potential annotation errors, this solution achieves a semi-automatic error correction mechanism of human-machine collaboration on the basis of accurate doctor-patient dialogue segmentation. This significantly reduces the workload of manual full proofreading while improving the final annotation accuracy rate, and reduces the risk of incorrect medical record attribution due to doctor's negligence or model misjudgment.

[0124] Example 3 Please see Figure 3 , Figure 3 This is another voice classification and processing system for doctor-patient dialogue analysis disclosed in the embodiments of the present invention. Figure 3 The described voice classification and processing system for doctor-patient dialogue analysis is applied in a data processing system / data processing equipment / data processing server (wherein, the server includes a local processing server or a cloud processing server). Figure 3 As shown, the speech classification and processing system for doctor-patient dialogue analysis may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the voice classification processing method for doctor-patient dialogue analysis described in Embodiment 1.

[0125] Example 4 This invention discloses a computer read storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the steps of the speech classification processing method for doctor-patient dialogue analysis described in Embodiment 1.

[0126] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the speech classification processing method for doctor-patient dialogue analysis described in Embodiment 1.

[0127] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0129] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0130] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0135] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0136] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0137] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0138] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0139] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0140] Finally, it should be noted that the voice classification processing method and system for doctor-patient dialogue analysis disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech classification processing method for analyzing doctor-patient dialogues, characterized in that, The method includes: Obtain voice data from doctor users; Based on the sound data, the voiceprint features of the doctor user are extracted; Acquire the voice data of the conversation between the doctor user and the patient user; The voiceprint features and the dialogue speech data are input into the trained speech recognition and segmentation model to obtain speech segmentation result data; the speech segmentation result data includes multiple speech text data labeled as the doctor user and the patient user, respectively.

2. The speech classification processing method for doctor-patient dialogue analysis according to claim 1, characterized in that, The speech recognition and segmentation model is trained through the following steps: Obtain a Chinese speech dataset; the Chinese speech dataset is the AISHELL-4 Chinese speech dataset publicly available by OpenSLR; The audio data in the Chinese speech dataset is preprocessed to obtain the processed data; Based on the processed data and the basic model of the ECAPA-TDNN architecture, the speech recognition and segmentation model is obtained through iterative training.

3. The speech classification processing method for doctor-patient dialogue analysis according to claim 2, characterized in that, The preprocessing includes at least one of the following processing operations: Perform resampling to the preset frequency; Mono conversion; Endpoint monitoring; And extract Mel spectrum features; the parameters of the Mel spectrum features include: 80 Mel frequency bands, 512-point FFT window, 160-point frame shift and 25ms window length.

4. The speech classification processing method for doctor-patient dialogue analysis according to claim 2, characterized in that, The iterative training includes: Using the x-vector model with 4.5% EER as the pre-trained model, the SE-Res2Net module and attention statistical pooling were introduced. Cross-entropy loss and label smoothing were used to enhance inter-class separation. SpecAugment, MUSAN noise injection, and RIR reverberation simulation were added to enhance the data, and the training was carried out for 100 epochs.

5. The speech classification processing method for doctor-patient dialogue analysis according to claim 1, characterized in that, The step of inputting the voiceprint features and the dialogue speech data into the trained speech recognition and segmentation model to obtain speech segmentation result data includes: The voiceprint features and each speech segment of the dialogue speech data are input into the trained speech recognition and segmentation model to obtain the speaker label corresponding to each of the output speech segments; All the speech segments are clustered using a spectral clustering algorithm to obtain multiple speech segment groups; The cosine similarity algorithm is used to determine whether adjacent groups of speech segments are spoken by the same person. If so, the annotation is updated to obtain the speaker annotation corresponding to each speech segment. Based on the speech-to-text model, the speech text corresponding to each speech segment is identified; Determine the corresponding speech text, speaker annotation, and timestamp annotation for each speech segment, and output the speech segmentation result data.

6. The speech classification processing method for doctor-patient dialogue analysis according to claim 5, characterized in that, The speech-to-text model is trained through the following steps: Zero-shot inference was performed using the Long From ASR model, which was trained on the AISHELL-4 test set. The encoder layers are set to 24, the first 12 layers of the encoder are frozen to retain general acoustic characteristics, and only the last 12 layers of the encoder and all decoders are fine-tuned. The learning rate was set to 1e-5, the label smoothing coefficient ε=0.1, and the speech-to-text model was obtained after 10 epochs of fine-tuning training.

7. The speech classification processing method for doctor-patient dialogue analysis according to claim 1, characterized in that, The acquisition of the doctor user's voice data includes: Based on the user parameters of the doctor users, match multiple historical doctor dialogue data with similar user parameters in the historical doctor-patient dialogue database; Based on the speech-to-text model, the historical dialogue text corresponding to each of the historical doctor dialogue data is determined. Based on the intersection calculation algorithm, the high-frequency text portion of all the historical dialogue texts is calculated to obtain the reference text; The guiding text, including the reference text, is played to the doctor user through a playback device so that the doctor user reads the reference text aloud, thereby obtaining the corresponding sound data; the voiceprint features are calculated through the following steps: Calculate the average value of the speech similarity between the sound data and each of the historical doctor dialogue data to obtain the corresponding speech similarity parameters; The sound data and the speech similarity parameters are input into the trained voiceprint extraction model to obtain the corresponding voiceprint features; the voiceprint extraction model is trained using a training dataset that includes multiple training sound data and corresponding historical speech similarity parameters and voiceprint feature annotations.

8. The speech classification processing method for doctor-patient dialogue analysis according to claim 5, characterized in that, The method further includes: In response to a correction instruction for the speech segmentation data, a speaker annotation correction operation is performed on the speech segmentation data; the speaker annotation correction operation includes: Acquire the doctor user's browsing operations on the speech segmentation data; Based on the operation recognition algorithm, the time points that belong to the hesitant operation in the browsing operation are identified, and multiple hesitant time points are obtained. Obtain the browsing audio segment corresponding to each hesitation time point in the speech segmentation data; For each of the browsing audio segments, calculate the first text coherence parameter between the audio text corresponding to the browsing audio segment and the audio text of the previous audio segment; Calculate the second text coherence parameter between the speech text corresponding to the browsed speech segment and the speech text of the previous speech segment; the first text coherence parameter and the second text coherence parameter are predicted by a trained text coherence prediction model; the text coherence prediction model is trained by a training dataset including multiple training speech text pairs and corresponding coherence annotations. Calculate the average value of the first text coherence parameter and the second text coherence parameter, and determine whether the average value of the parameters is greater than a preset parameter threshold; If so, the speaker tag corresponding to the browsed audio segment is determined as the one that appears most frequently in the nearby tag set; the nearby tag set is the set of speaker tags corresponding to the browsed audio segment before and after a preset number of the browsed audio segment. If not, the speaker tag corresponding to the viewed audio segment is determined to be the one that appears least frequently in the nearby tag set.

9. A speech classification and processing system for analyzing doctor-patient dialogue, characterized in that, The system includes: The acquisition module is used to acquire the voice data of doctor users; The extraction module is used to extract the voiceprint features of the doctor user based on the sound data; The acquisition module is used to acquire dialogue voice data, including the conversation between the doctor user and the patient user; The recognition module is used to input the voiceprint features and the dialogue speech data into a trained speech recognition and segmentation model to obtain speech segmentation result data; the speech segmentation result data includes multiple speech text data labeled as the doctor user and the patient user, respectively.

10. A speech classification and processing system for analyzing doctor-patient dialogue, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the speech classification processing method for doctor-patient dialogue analysis as described in any one of claims 1-8.

Images