A specialist medical record robot system and method based on a vertical field large model

Abstract

The application relates to the technical field of intelligent medical treatment and natural language processing, and discloses a special medical record robot system and method based on a vertical field large model, the method comprising the following steps: an adaptive spectral subtraction algorithm with a spectral lower limit is used in a preprocessing and noise reduction module to filter out background noise in dialogue voice signals to generate clear voice signals; a cross-modal language processing module extracts target medical sub-field labels and generates a temporary hot word set according to a context, and the temporary hot word set is reversely input into a dynamic speech recognition module to adjust a decoding graph path; the cross-modal language processing module combines semantic confidence scores and acoustic sequence metadata to jointly evaluate medical key information and assign an intention label; a medical record generation module fills medical key information with the intention label into corresponding slots of an outpatient medical record template according to the intention label to generate an outpatient medical record draft; and the application effectively filters out environmental noise, improves medical vocabulary recognition accuracy, and accurately distinguishes between confirmed and suspected signs.

Description

Technical Field

[0001] This invention relates to the fields of intelligent medical care and natural language processing technology, specifically to a specialized medical record robot system and method based on a large vertical domain model. Background Technology

[0002] In recent years, with the application of artificial intelligence technology in the medical field, using speech recognition and natural language processing technology to assist doctors in automatically generating electronic medical records has become an important means of improving outpatient efficiency. However, existing medical record generation systems mainly face the following technical shortcomings in practical clinical applications: Real hospital outpatient clinics and physical spaces are often noisy, with a lot of background noise from human voices and equipment operation noise. Most existing voice acquisition devices use conventional hardware filtering or basic noise reduction algorithms, which cannot effectively separate effective human voices from background noise in complex sound fields. This results in the voice signal entering the system being mixed with a lot of noise, which directly reduces the input quality and resolution of the front-end acoustic decoding.

[0003] In specialized medical settings, there are a large number of obscure medical terms and professional terms. Conventional speech recognition models are mostly trained on general corpora or only use static medical dictionaries for matching. When facing doctor-patient dialogues, existing systems cannot dynamically perceive the medical subdomain according to the current context and lack real-time hot word feedback and path adjustment mechanisms. As a result, the system has a low accuracy rate in recognizing obscure specialized words and professional terms, and doctors need to do a lot of manual verification and modification later.

[0004] Clinical medical records demand a high degree of objectivity and rigor, requiring a strict distinction between a patient's confirmed condition and suspected symptoms. Existing medical record extraction modules generally rely solely on a single text modality for semantic information extraction, neglecting the acoustic characteristics of the patient or doctor's presentation. Due to the lack of a joint evaluation mechanism that combines acoustic sequence features with text semantics, existing systems struggle to accurately determine the degree of diagnosis of the presented content. This often results in the final draft medical record confusing confirmed facts with suspected cases, failing to accurately fill the correct slots in the medical record template, and thus not conforming to the recording standards for objective clinical diagnosis. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides the following technical solution: a specialized medical record robot system based on a vertical domain large model, wherein the specialized medical record robot system based on a vertical domain large model is deployed in a local computing device, including a local model deployment module configured in the local computing device, and a voice acquisition module, a preprocessing and noise reduction module, a dynamic speech recognition module, a cross-modal language processing module, a medical record generation module, an interactive display module, and a storage and interface module that are interconnected. The local model deployment module is communicatively connected to the dynamic speech recognition module and the cross-modal language processing module, respectively. The local model deployment module is used to load and run the speech recognition model and the natural language processing model; the dynamic speech recognition module calls the speech recognition model, and the cross-modal language processing module calls the natural language processing model. The voice acquisition module is used to acquire dialogue voice signals and transmit the dialogue voice signals to the preprocessing and noise reduction module; The preprocessing and noise reduction module is used to perform time-frequency domain conversion and noise suppression operations on the dialogue speech signal, filter out background noise to generate a clear speech signal, and transmit the clear speech signal to the dynamic speech recognition module. The dynamic speech recognition module is used to receive the clear speech signal and perform acoustic decoding operations. The acoustic decoding operations generate a dialogue text sequence based on a decoding graph search path. The dynamic speech recognition module simultaneously extracts the acoustic sequence metadata corresponding to the dialogue text sequence, combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, and transmits it to the cross-modal language processing module. The dynamic speech recognition module is also used to continuously output intermediate recognized text to the cross-modal language processing module during the acoustic decoding operation. The cross-modal language processing module pre-stores multiple medical subdomain labels, and the intermediate recognition text contains contextual data. The cross-modal language processing module is used to extract the contextual data, calculate the predicted probability of each medical subdomain label based on the contextual data, extract the target medical subdomain label, generate a temporary hot word set based on the target medical subdomain label, and input the temporary hot word set back into the dynamic speech recognition module to instruct the dynamic speech recognition module to adjust the decoding graph search path. The cross-modal language processing module is further configured to receive the multimodal data stream and perform medical entity extraction operations, outputting key medical information; the key medical information includes text content; the cross-modal language processing module calculates a semantic confidence score based on the text content, and performs a joint evaluation of the key medical information by combining the semantic confidence score with the acoustic sequence metadata, calculating a diagnostic score; the cross-modal language processing module assigns a corresponding intent tag to the key medical information according to the diagnostic score, and transmits the key medical information with the intent tag to the medical record generation module; The medical record generation module has a pre-stored outpatient medical record template with multiple slots. The medical record generation module is used to fill the medical key information with the intent tag into the corresponding slots of the multiple slots of the outpatient medical record template according to the intent tag, generate an outpatient medical record draft, and transmit the outpatient medical record draft to the interactive display module. The interactive display module is used to receive operation instructions for the draft outpatient medical record, generate a confirmed outpatient medical record, and transmit the confirmed outpatient medical record to the storage and interface module. The storage and interface module is configured with a data interface, which is used to store the confirmed outpatient medical records to the local computing device and transmit the confirmed outpatient medical records to an external hospital information system through the data interface.

[0006] Preferably, the preprocessing and noise reduction module performs time-frequency domain conversion on the dialogue speech signal to generate a noisy speech spectrum sequence. The noisy speech spectrum sequence contains multiple noisy speech spectrum frames, each with a corresponding amplitude. The multiple noisy speech spectrum frames include a target noisy speech spectrum frame. The preprocessing and noise reduction module is configured with a preset energy threshold. The preprocessing and noise reduction module performs endpoint detection on the noisy speech spectrum sequence, calculates the short-time energy of each noisy speech spectrum frame, compares each short-time energy with the preset energy threshold, marks noisy speech spectrum frames with short-time energy lower than the preset energy threshold as non-speech segments, and calculates the estimated spectral amplitude of the background noise corresponding to each noisy speech spectrum frame based on the non-speech segments. The preprocessing and noise reduction module is configured with a preset adaptive over-subtraction factor and a preset lower spectral limit constant. The preprocessing and noise reduction module uses an adaptive spectral subtraction algorithm with a lower spectral limit to process the noisy speech spectrum sequence and calculate the spectrum estimate of the clean speech. The spectrum estimate of the clean speech is obtained by comparing a first operation value and a second operation value and taking the maximum value. The first operation value is the amplitude corresponding to the target noisy speech spectrum frame, minus the product of the preset adaptive over-subtraction factor and the spectrum amplitude estimate of the background noise corresponding to the target noisy speech spectrum frame. The second operation value is the product of the preset lower spectral limit constant and the amplitude corresponding to the target noisy speech spectrum frame.

[0007] Preferably, the cross-modal language processing module is configured with a sliding window, and the cross-modal language processing module pre-stores probability judgment thresholds; the natural language processing model is configured with fine-tuning parameters; The cross-modal language processing module receives the intermediate recognized text in real time through the sliding window to extract the contextual data contained in the intermediate recognized text; it performs an autoregressive prediction operation on the contextual data based on the fine-tuning parameters of the natural language processing model to calculate the prediction probability of each medical subdomain label; the cross-modal language processing module extracts medical subdomain labels whose prediction probabilities exceed the probability judgment threshold as the target medical subdomain labels; the temporary hot word set contains multiple candidate medical hot words, the cross-modal language processing module calculates the prior probability of each candidate medical hot word, and inputs the temporary hot word set and the prior probabilities of each candidate medical hot word in reverse to the dynamic speech recognition module.

[0008] Preferably, the dynamic speech recognition module is equipped with a basic medical hot word dictionary; The dynamic speech recognition module receives the temporary hot word set and the prior probabilities of each candidate medical hot word, and calculates the fusion weight of each candidate medical hot word. The fusion weight is the sum of the basic static weight and the dynamic feedback adjustment term. The basic static weight is the weight assigned to the candidate medical hot word by the basic medical hot word dictionary. The dynamic feedback adjustment term is the product of a preset dynamic feedback adjustment coefficient, the prior probability of the candidate medical hot word, and a time decay factor. The time decay factor is determined based on a preset time decay constant and a time difference, where the time difference is the time difference between the current operation time and the trigger time when the cross-modal language processing module calculates the prior probability. The dynamic speech recognition module calculates the acoustic log-likelihood score corresponding to each candidate medical hot word in the decoding graph search path, injects each fusion weight into the decoding graph search path, and superimposes each fusion weight into the acoustic log-likelihood score of the corresponding candidate medical hot word; the dynamic speech recognition module updates the value of each fusion weight in real time according to the time decay factor, and adjusts the node retention status of each candidate medical hot word in the decoding graph search path according to the updated fusion weight.

[0009] Preferably, the dialogue text sequence contains multiple lexical units; the acoustic sequence metadata includes acoustic confidence scores and pronunciation pause features; The dynamic speech recognition module synchronously extracts the acoustic sequence metadata corresponding to the dialogue text sequence, and combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, specifically including: The dynamic speech recognition module extracts the acoustic sequence metadata corresponding to each word in the dialogue text sequence, aligns the dialogue text sequence with the acoustic sequence metadata in the time dimension, and combines them into the multimodal data stream.

[0010] Preferably, the cross-modal language processing module identifies the target medical entity in the dialogue text sequence through the natural language processing model, extracts the target medical entity in the dialogue text sequence to generate the key medical information, and records the key medical information as a medical entity sequence; The medical entity sequence contains the corresponding target words in the dialogue text sequence, and the medical entity sequence has a sequence position and a pronunciation preposition position. The acoustic sequence metadata has a data structure. The cross-modal language processing module aligns the medical entity sequence with the acoustic sequence metadata using multimodal features, compares the sequence position with the data structure, extracts the acoustic confidence score in the acoustic sequence metadata corresponding to each target word, and calculates the average acoustic confidence score of all target words contained in the medical entity sequence. The cross-modal language processing module extracts the pronunciation pause features in the acoustic sequence metadata corresponding to each target word, and identifies the pronunciation preposition position and the total duration of pronunciation pauses within all target words based on the pronunciation pause features.

[0011] Preferably, the cross-modal language processing module pre-stores a mapping table of limited modifiers and score weights, as well as a medical judgment threshold, and is configured with a first normalized weight coefficient, a second normalized weight coefficient, and a third normalized weight coefficient; the medical entity sequence has contextual content; the cross-modal language processing module extracts the contextual content, compares the contextual content with the mapping table, and extracts the corresponding score weight as the semantic confidence score based on the mapping table; The diagnostic score calculated by the cross-modal language processing module is the difference between the sum of the acoustic weighting term and the semantic weighting term and the pause penalty term; the acoustic weighting term is the product of the average acoustic confidence score and the first normalized weighting coefficient, the pause penalty term is the product of the total duration of the pronunciation pause and the second normalized weighting coefficient, and the semantic weighting term is the product of the semantic confidence score and the third normalized weighting coefficient. The cross-modal language processing module compares the diagnostic score with the medical judgment threshold. When the diagnostic score is greater than or equal to the medical judgment threshold, it assigns a confirmed category intent label to the medical entity sequence; when the diagnostic score is less than the medical judgment threshold, it assigns a suspected category intent label to the medical entity sequence.

[0012] Preferably, the outpatient medical record template includes multiple slots such as chief complaint slot, present illness history slot, past medical history slot, physical signs slot, diagnostic opinion slot, medication recommendation slot, and examination item slot; the outpatient medical record template is configured with confirmed sub-slots and suspected sub-slots under the diagnostic opinion slot and the medication recommendation slot; The medical key information with the intent tag includes medical entity content and the intent tag, and the medical entity content has corresponding attributes; the medical record generation module extracts the medical entity content and the intent tag contained in the medical key information with the intent tag, matches the corresponding slot in the plurality of slots according to the corresponding attributes of the medical entity content, and further matches the confirmed sub-slot or the suspected sub-slot according to the category of the intent tag for filling.

[0013] Preferably, the local computing device is configured with an external input device, and the interactive display module receives the operation instructions transmitted by the external input device. The operation instructions include editing data and confirmation data for the draft outpatient medical record. The draft outpatient medical record includes draft text content. The interactive display module updates the draft text content of the draft outpatient medical record according to the operation instructions and generates the confirmed outpatient medical record. The data interface is an encrypted data transmission interface; the storage and interface module receives a data synchronization instruction, performs a data encryption operation on the confirmed outpatient medical record according to the data synchronization instruction, and transmits the encrypted confirmed outpatient medical record to the external hospital information system through the encrypted data transmission interface.

[0014] Preferably, a specialist medical record robot method based on a vertical domain large model is applied to the aforementioned specialist medical record robot system based on a vertical domain large model. The specialist medical record robot method based on a vertical domain large model includes the following steps: S1. The voice acquisition module acquires the dialogue voice signal and transmits it to the preprocessing and noise reduction module; the preprocessing and noise reduction module performs time-frequency domain conversion and noise suppression operations on the dialogue voice signal, filters out background noise and generates a clear voice signal, and transmits the clear voice signal to the dynamic voice recognition module. S2. The dynamic speech recognition module receives the clear speech signal and performs acoustic decoding. The acoustic decoding operation generates a dialogue text sequence based on the decoding graph search path, and continuously outputs intermediate recognized text to the cross-modal language processing module during the acoustic decoding operation. The cross-modal language processing module pre-stores multiple medical subdomain tags. The intermediate recognized text contains contextual data. The cross-modal language processing module extracts the contextual data, calculates the predicted probability of each medical subdomain tag based on the contextual data, extracts the target medical subdomain tag, and generates a temporary hot word set based on the target medical subdomain tag. The temporary hot word set is then input back to the dynamic speech recognition module. The dynamic speech recognition module adjusts the decoding graph search path based on the temporary hot word set. S3. When generating the dialogue text sequence, the dynamic speech recognition module simultaneously extracts the acoustic sequence metadata corresponding to the dialogue text sequence, combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, and transmits it to the cross-modal language processing module. S4. The cross-modal language processing module receives the multimodal data stream and performs medical entity extraction, outputting key medical information. The key medical information includes text content. The cross-modal language processing module calculates a semantic confidence score based on the text content. The cross-modal language processing module combines the semantic confidence score with the acoustic sequence metadata to jointly evaluate the key medical information and calculate a diagnostic accuracy score. The cross-modal language processing module assigns a corresponding intent tag to the key medical information according to the diagnostic accuracy score and transmits the key medical information with the intent tag to the medical record generation module. S5. The medical record generation module pre-stores an outpatient medical record template with multiple slots. The medical record generation module fills the key medical information with the intent tag into the corresponding slots of the multiple slots of the outpatient medical record template according to the intent tag, generates an outpatient medical record draft, and transmits it to the interactive display module. The interactive display module receives operation instructions for the outpatient medical record draft, generates a confirmed outpatient medical record, and transmits the confirmed outpatient medical record to the storage and interface module. The storage and interface module is configured with a data interface. The storage and interface module receives the confirmed outpatient medical record and stores it in the local computing device, and transmits the confirmed outpatient medical record to an external hospital information system through the data interface.

[0015] This invention provides a specialized medical record robot system and method based on a large vertical domain model. It has the following beneficial effects: (1) This invention uses an adaptive spectral subtraction algorithm with a lower spectral limit to process the noisy speech spectrum sequence through a preprocessing and noise reduction module, and then calculates the spectrum estimate of the clean speech. This can effectively filter out background noise in the dialogue speech signal, thereby generating a clear speech signal. This avoids the interference of real environmental noise in the hospital on the front-end sound acquisition, and provides a high-quality input basis for subsequent acoustic decoding.

[0016] (2) In this invention, the cross-modal language processing module calculates the predicted probability of medical subdomain labels based on the context of the intermediate recognized text, extracts the target medical subdomain labels, and then generates a temporary hot word set based on the target medical subdomain labels. By inputting the temporary hot word set back into the dynamic speech recognition module, the dynamic speech recognition module adjusts the decoding graph path based on the temporary hot word set, which effectively improves the recognition accuracy of professional terms and obscure medical terms in specialized medical scenarios.

[0017] (3) The cross-modal language processing module of the present invention combines semantic confidence score and acoustic sequence metadata to jointly evaluate medical key information, thereby calculating the diagnostic score, and assigning corresponding intent tags to medical key information according to the diagnostic score. The medical record generation module fills the medical key information with intent tags into the corresponding slots of the outpatient medical record template according to the intent tags. This process distinguishes between confirmed and suspected signs from a multimodal perspective, ensuring that the final output outpatient medical record meets the objective logic and accuracy requirements of clinical diagnosis. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system framework of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See attached document Figure 1 This invention provides a specialized medical record robot system based on a large vertical domain model. The specialized medical record robot system based on a large vertical domain model includes a voice acquisition module, a preprocessing and noise reduction module, a local model deployment module, a dynamic speech recognition module, a cross-modal language processing module, a medical record generation module, an interactive display module, and a storage and interface module.

[0021] The specialized medical record robot system based on a large vertical domain model is deployed on a local computing device. The local model deployment module is configured on the local computing device and communicates with the dynamic speech recognition module and the cross-modal language processing module. The local model deployment module is used to load and run the speech recognition model called by the dynamic speech recognition module and the natural language processing model called by the cross-modal language processing module.

[0022] The voice acquisition module communicates with the preprocessing and noise reduction module; the preprocessing and noise reduction module communicates with the dynamic speech recognition module; the dynamic speech recognition module communicates bidirectionally with the cross-modal language processing module; the cross-modal language processing module communicates with the medical record generation module; the medical record generation module communicates with the interactive display module; and the interactive display module communicates with the storage and interface module.

[0023] See attached document Figure 2 The present invention also provides a specialist medical record robot method based on a vertical domain large model. This method is applied to the aforementioned specialist medical record robot system based on a vertical domain large model. The specialist medical record robot method based on a vertical domain large model includes the following steps: S1. The voice acquisition module acquires the dialogue voice signal and transmits it to the preprocessing and noise reduction module. The preprocessing and noise reduction module performs time-frequency domain conversion and noise suppression on the dialogue voice signal. The preprocessing and noise reduction module filters out background noise in the dialogue voice signal and generates a clear voice signal. The preprocessing and noise reduction module transmits the clear voice signal to the dynamic speech recognition module.

[0024] S2. The dynamic speech recognition module receives a clear speech signal and performs acoustic decoding. During the acoustic decoding process, it continuously outputs intermediate recognized text. The cross-modal language processing module receives the intermediate recognized text output by the dynamic speech recognition module. The cross-modal language processing module calculates the predicted probability of the medical subdomain label based on the context of the intermediate recognized text and extracts the target medical subdomain label. The cross-modal language processing module generates a temporary hot word set based on the target medical subdomain label. The cross-modal language processing module inputs the temporary hot word set back to the dynamic speech recognition module. The dynamic speech recognition module adjusts the decoding graph path based on the temporary hot word set and generates a dialogue text sequence.

[0025] S3. When generating the dialogue text sequence, the dynamic speech recognition module simultaneously extracts the acoustic sequence metadata corresponding to the dialogue text sequence. The acoustic sequence metadata includes acoustic confidence scores and pronunciation pause features. The dynamic speech recognition module combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, and then transmits the multimodal data stream to the cross-modal language processing module.

[0026] S4. The cross-modal language processing module receives multimodal data streams and performs medical entity extraction operations. The cross-modal language processing module outputs key medical information, extracts acoustic sequence metadata corresponding to the key medical information, calculates semantic confidence scores based on the text content of the key medical information, performs joint evaluation of the key medical information by combining the semantic confidence scores and acoustic sequence metadata, calculates diagnostic accuracy scores, assigns corresponding intent tags to the key medical information according to the diagnostic accuracy scores, outputs key medical information with intent tags, and transmits the key medical information with intent tags to the medical record generation module.

[0027] S5. The medical record generation module receives key medical information with intent tags. The medical record generation module has a pre-stored outpatient medical record template. The medical record generation module fills the key medical information with intent tags into the corresponding slots of the outpatient medical record template according to the intent tags. The medical record generation module generates an outpatient medical record draft. The medical record generation module transmits the outpatient medical record draft to the interactive display module. The interactive display module receives operation instructions. The interactive display module generates a confirmed outpatient medical record. The interactive display module transmits the confirmed outpatient medical record to the storage and interface module. The storage and interface module stores the confirmed outpatient medical record to the local terminal. The storage and interface module transmits the confirmed outpatient medical record to the external hospital information system through the data interface.

[0028] In step S1, the voice acquisition module includes a microphone array. The voice acquisition module acquires the original sound field signal in the physical space through the microphone array. The voice acquisition module performs analog-to-digital conversion on the original sound field signal and generates a dialogue voice signal. The dialogue voice signal is superimposed with environmental noise signal.

[0029] The voice acquisition module transmits the dialogue voice signal to the preprocessing and noise reduction module. The preprocessing and noise reduction module performs framing and windowing operations on the dialogue voice signal. The preprocessing and noise reduction module divides the dialogue voice signal into multiple short voice frames. The preprocessing and noise reduction module performs short-time Fourier transform operations on multiple short voice frames. The preprocessing and noise reduction module converts the dialogue voice signal in the time domain into a noisy voice spectrum sequence in the frequency domain.

[0030] The preprocessing and noise reduction module performs endpoint detection on the noisy speech spectrum sequence. It calculates the short-time energy of each frame in the noisy speech spectrum sequence, compares the short-time energy with a set energy threshold, and marks frames with short-time energy below the energy threshold as non-speech segments. Based on the non-speech segments, the preprocessing and noise reduction module calculates the estimated spectral amplitude of the background noise. Finally, it selects the frame with the short-time energy below the energy threshold from the noisy speech spectrum sequence. Frame and angular frequency is The amplitude of the noisy speech spectrum is denoted as The preprocessing and noise reduction module will work with the first... The estimated spectral amplitude of the background noise corresponding to the frame is denoted as .

[0031] The preprocessing and noise reduction module uses an adaptive spectral subtraction algorithm with a spectral lower bound to process the noisy speech spectrum sequence, and calculates the spectral estimate of the clean speech. Spectral estimation of clean speech The calculation formula is:

[0032] In the above calculation formula, The frame number representing the short speech frame. The angular frequency representing the speech spectrum. Indicates the first spectral sequence of noisy speech. Frame and angular frequency is Spectral estimates of clean speech, Indicates the first spectral sequence of noisy speech. Frame and angular frequency is The amplitude of the noisy speech spectrum. Indicates the relationship with the first The estimated spectral amplitude of the background noise corresponding to the frame. Indicates the adaptive over-subtraction factor. Indicates the lower limit constant of the spectrum. This represents the function that takes the maximum value.

[0033] The preprocessing and denoising module calculates the prior signal-to-noise ratio (SNR) of the noisy speech spectrum sequence, and adjusts the adaptive over-subtraction factor based on the prior SNR. The numerical values ​​are processed by the preprocessing and noise reduction module through an adaptive over-subtraction factor. Adjusting the noise suppression intensity, the preprocessing and noise reduction module uses the lower limit constant of the spectrum. Control the lower limit amplitude of the spectrum of noisy speech.

[0034] The preprocessing and noise reduction module uses the spectral estimation value of the clean speech. A clean speech spectrum sequence is generated. The preprocessing and noise reduction module performs an inverse short-time Fourier transform operation on the clean speech spectrum sequence. The preprocessing and noise reduction module restores the clean speech spectrum sequence in the frequency domain to a time domain signal. The preprocessing and noise reduction module generates a clear speech signal and transmits the clear speech signal to the dynamic speech recognition module.

[0035] In step S2, the dynamic speech recognition module is equipped with a basic medical hot word dictionary. The dynamic speech recognition module receives a clear speech signal and performs an acoustic decoding operation based on the decoding graph search path on the clear speech signal. It also calculates the acoustic log-likelihood score of each candidate word in the decoding graph search path. During the acoustic decoding operation, the dynamic speech recognition module continuously extracts decoding features and outputs intermediate recognition text. The dynamic speech recognition module then transmits the intermediate recognition text to the cross-modal language processing module.

[0036] The cross-modal language processing module is equipped with a sliding window. It receives intermediate recognized text in real time through this window, extracts contextual data from the intermediate recognized text, and pre-stores multiple medical subdomain labels. Based on the fine-tuned parameters of the natural language processing model it calls, the module performs autoregressive prediction on the contextual data, calculates the predicted probability of each medical subdomain label, and pre-stores probability judgment thresholds. It extracts medical subdomain labels whose predicted probabilities exceed the probability judgment thresholds as target medical subdomain labels. The module generates a temporary hot word set based on the target medical subdomain labels, calculates the prior probability of each candidate medical hot word in the temporary hot word set, and then inputs the temporary hot word set and the prior probabilities of each candidate medical hot word back into the dynamic speech recognition module.

[0037] The dynamic speech recognition module receives a temporary hot word set and the prior probabilities of each candidate medical hot word. It then calculates the fusion weight of each candidate medical hot word in the temporary hot word set and records the candidate medical hot words as follows: The dynamic speech recognition module will select candidate medical hot words. The fusion weight is denoted as Candidate medical buzzwords Fusion weights The calculation formula is:

[0038] In the above calculation formula, This indicates candidate medical hot words in the temporary hot word set. Indicating candidate medical hot words The fusion weight, This indicates that the basic medical hot words dictionary assigns candidate medical hot words. The basic static weights, This represents the preset dynamic feedback adjustment coefficient. This represents the contextual data corresponding to the intermediate recognized text extracted by the cross-modal language processing module. This indicates that the cross-modal language processing module is based on contextual data. Calculated candidate medical hot words The prior probability of occurrence, Represents the natural constant. This represents the preset time decay constant. This represents the time difference between the current computation time and the trigger time when the cross-modal language processing module generates the prior probability. This represents the time decay factor.

[0039] The dynamic speech recognition module will integrate weights By injecting a decoding graph search path, the dynamic speech recognition module will integrate weights. Overlay to candidate medical hot words In the corresponding acoustic log-likelihood score, the dynamic speech recognition module uses the time decay factor... Real-time updates of fusion weights The dynamic speech recognition module adjusts the values ​​based on the updated fusion weights. Adjusting candidate medical keywords in the decoding graph search path The node retains its state, and the dynamic speech recognition module completes the acoustic decoding of the clear speech signal based on the adjusted decoding graph search path. The dynamic speech recognition module then generates a dialogue text sequence. In step S3, for the generated dialogue text sequence, the dynamic speech recognition module synchronously extracts the acoustic sequence metadata corresponding to each word in the dialogue text sequence. The acoustic sequence metadata includes acoustic confidence scores and pronunciation pause features. The dynamic speech recognition module aligns the dialogue text sequence and the acoustic sequence metadata in the time dimension and combines them into a multimodal data stream. The dynamic speech recognition module transmits the multimodal data stream to the cross-modal language processing module.

[0040] Furthermore, in step S4, the cross-modal language processing module receives the multimodal data stream output by the dynamic speech recognition module. The multimodal data stream includes a dialogue text sequence and acoustic sequence metadata corresponding to the dialogue text sequence.

[0041] The cross-modal language processing module invokes a natural language processing model to process the dialogue text sequence. The module identifies target medical entities within the dialogue text sequence using the natural language processing model. These target medical entities include the chief complaint, present illness, past medical history, physical signs, diagnostic opinions, medication recommendations, and examination items. The module extracts these target medical entities from the dialogue text sequence and generates key medical information. Finally, the module records this key medical information as a sequence of medical entities. .

[0042] The cross-modal language processing module will process medical entity sequences. Multimodal feature alignment with acoustic sequence metadata, and comparison of medical entity sequences by a cross-modal language processing module. The sequence position and data structure of the acoustic sequence metadata are used to extract the medical entity sequence from the acoustic sequence metadata by the cross-modal language processing module. Acoustic confidence scores, cross-modal language processing module calculates medical entity sequences The average acoustic confidence score of all included lexical units, denoted as .

[0043] The cross-modal language processing module extracts the medical entity sequence from the acoustic sequence metadata. Pronunciation pause features, cross-modal language processing module recognizes medical entity sequences The position of pronunciation before the sound and the total duration of pronunciation pauses within the word are recorded as follows: .

[0044] The cross-modal language processing module computes medical entity sequences. The semantic confidence score is denoted as . The cross-modal language processing module has a pre-stored mapping table between qualifying modifiers and score weights. The cross-modal language processing module compares medical entity sequences. The context content and mapping table are used by the cross-modal language processing module to extract the corresponding score weights as semantic confidence scores. The cross-modal language processing module performs cross-modal feature fusion calculation operations and calculates medical entity sequences. The diagnostic accuracy score is denoted as... Diagnostic score The calculation formula is:

[0045] In the above calculation formula, This represents the extracted medical entity sequence. Represents medical entity sequence Diagnostic score, Represents medical entity sequence The average acoustic confidence score. Represents medical entity sequence The corresponding total duration of pronunciation pauses, Represents medical entity sequence The semantic confidence score, This represents the normalized weighting coefficient corresponding to the average acoustic confidence score. This represents the normalized weighting coefficient corresponding to the total duration of the pronunciation pause. This represents the normalized weight coefficient corresponding to the semantic confidence score.

[0046] The cross-modal language processing module has pre-stored medical judgment thresholds. The cross-modal language processing module compares diagnostic scores. Medical judgment threshold The value when the diagnostic score is... Greater than or equal to the medical judgment threshold At that time, the cross-modal language processing module processed medical entity sequences. The intention label for assigning confirmed case categories, when the positivity score is... Less than the medical judgment threshold At that time, the cross-modal language processing module processed medical entity sequences. Assign intent labels to suspected categories.

[0047] The cross-modal language processing module combines intent tags with medical entity sequences. The cross-modal language processing module performs associative encapsulation, outputs key medical information with intent tags, and then transmits the key medical information with intent tags to the medical record generation module.

[0048] In step S5, the medical record generation module receives key medical information with intent tags transmitted by the cross-modal language processing module. The medical record generation module has a pre-stored outpatient medical record template. The outpatient medical record template contains multiple corresponding slots, including the chief complaint slot, present illness slot, past medical history slot, physical signs slot, diagnostic opinion slot, medication recommendation slot, and examination item slot. The outpatient medical record template has confirmed sub-slots and suspected sub-slots under the diagnostic opinion slot and the medication recommendation slot.

[0049] The medical record generation module parses key medical information with intent tags, extracts the medical entity content and corresponding intent tags contained in the key medical information with intent tags, and matches the chief complaint slot, present illness slot, past medical history slot, physical signs slot, diagnosis opinion slot, medication recommendation slot or examination item slot of the outpatient medical record template according to the attributes of the medical entity content. The medical record generation module further matches confirmed sub-slots and suspected sub-slots according to the category of intent tags.

[0050] When the intent label is a confirmed case, the medical record generation module fills the confirmed case sub-slot of the outpatient medical record template with the key medical information with the intent label. When the intent label is a suspected case, the medical record generation module fills the suspected case sub-slot of the outpatient medical record template with the key medical information with the intent label. The medical record generation module performs the corresponding slot filling operation of the outpatient medical record template and generates an outpatient medical record draft.

[0051] The medical record generation module transmits the draft outpatient medical record to the interactive display module. The interactive display module generates an interactive interface and displays the draft outpatient medical record through the interactive interface. The local computing device is equipped with an external input device. The interactive display module receives operation instructions transmitted from the external input device. The operation instructions include editing data and confirmation data for the draft outpatient medical record.

[0052] The interactive display module updates the text content of the draft outpatient medical record according to the operation instructions. The interactive display module generates a confirmed outpatient medical record and transmits the confirmed outpatient medical record to the storage and interface module. The storage and interface module receives the confirmed outpatient medical record and stores it in the storage medium of the local terminal.

[0053] The storage and interface module is equipped with a data transmission interface, which is an encrypted data transmission interface. The storage and interface module connects to the external hospital information system through the encrypted data transmission interface. The storage and interface module receives data synchronization instructions, performs data encryption operations on the confirmed outpatient medical records according to the data synchronization instructions, and transmits the encrypted confirmed outpatient medical records to the external hospital information system through the encrypted data transmission interface.

[0054] The present invention also provides an electronic device, corresponding to the aforementioned local computing device. The electronic device includes a processor and a memory communicatively connected to the processor. The memory stores computer execution instructions, and the processor executes the aforementioned specialized medical record robot method based on a large vertical domain model.

[0055] Electronic devices also include input interfaces, display interfaces, and communication interfaces. Electronic devices include a system bus. Processors, memory, input interfaces, display interfaces, and communication interfaces are all connected through the system bus. The system bus provides a physical channel for data transmission between various hardware components. The processor includes a central processing unit and a graphics processing unit. The central processing unit is responsible for basic logic operations, and the graphics processing unit is responsible for parallel matrix operations of the speech recognition model and the natural language processing model.

[0056] The input interface is used to connect to external input devices. The processor receives operation instructions through the input interface. The display interface is used to connect to external display devices. The processor outputs an interactive interface through the display interface. The communication interface corresponds to the encrypted data transmission interface mentioned above. The processor interfaces with the external hospital information system through the communication interface.

[0057] The memory includes non-volatile storage media and internal memory. Non-volatile storage media include read-only memory, disks, or solid-state drives. The non-volatile storage media stores the operating system and computer execution instructions. The non-volatile storage media loads the operating system and computer execution instructions into the internal memory, and the processor reads the computer execution instructions from the internal memory.

[0058] When the processor executes computer instructions, it instantiates the logic operation modules included in the specialized medical record robot system based on a large vertical domain model in the internal memory. The processor instantiates the voice acquisition module, preprocessing and noise reduction module, local model deployment module, dynamic speech recognition module, cross-modal language processing module, medical record generation module, interactive display module, and storage and interface module. The processor completes the steps from dialogue voice signal processing to confirmation of outpatient medical record generation by calling the above modules.

[0059] The present invention also provides a computer-readable storage medium, which is a non-volatile data storage carrier. The computer-readable storage medium includes a USB flash drive, a portable hard drive, a read-only memory, or a magnetic disk. The computer-readable storage medium stores a computer program. When the computer program is loaded and executed by a processor, it implements the steps and operations included in the above-mentioned specialized medical record robot method based on a large vertical domain model.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A specialized medical record robot system based on a large vertical domain model, characterized in that, The specialized medical record robot system based on a large vertical domain model is deployed in a local computing device, including a local model deployment module configured in the local computing device, and a voice acquisition module, a preprocessing and noise reduction module, a dynamic speech recognition module, a cross-modal language processing module, a medical record generation module, an interactive display module, and a storage and interface module that are interconnected. The local model deployment module is communicatively connected to the dynamic speech recognition module and the cross-modal language processing module, respectively. The local model deployment module is used to load and run the speech recognition model and the natural language processing model; the dynamic speech recognition module calls the speech recognition model, and the cross-modal language processing module calls the natural language processing model. The voice acquisition module is used to acquire dialogue voice signals and transmit the dialogue voice signals to the preprocessing and noise reduction module; The preprocessing and noise reduction module is used to perform time-frequency domain conversion and noise suppression operations on the dialogue speech signal, filter out background noise to generate a clear speech signal, and transmit the clear speech signal to the dynamic speech recognition module. The dynamic speech recognition module is used to receive the clear speech signal and perform acoustic decoding operations, which generate a dialogue text sequence based on the decoding graph search path. The dynamic speech recognition module synchronously extracts the acoustic sequence metadata corresponding to the dialogue text sequence, combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, and transmits it to the cross-modal language processing module. The dynamic speech recognition module is also used to continuously output intermediate recognized text to the cross-modal language processing module during the acoustic decoding operation. The cross-modal language processing module pre-stores multiple medical subdomain labels, and the intermediate recognition text contains contextual data. The cross-modal language processing module is used to extract the contextual data, calculate the predicted probability of each medical subdomain label based on the contextual data, extract the target medical subdomain label, generate a temporary hot word set based on the target medical subdomain label, and input the temporary hot word set back into the dynamic speech recognition module to instruct the dynamic speech recognition module to adjust the decoding graph search path. The cross-modal language processing module is further configured to receive the multimodal data stream and perform medical entity extraction operations, outputting key medical information; the key medical information includes text content; the cross-modal language processing module calculates a semantic confidence score based on the text content, and performs a joint evaluation of the key medical information by combining the semantic confidence score with the acoustic sequence metadata, calculating a diagnostic score; the cross-modal language processing module assigns a corresponding intent tag to the key medical information according to the diagnostic score, and transmits the key medical information with the intent tag to the medical record generation module; The medical record generation module has a pre-stored outpatient medical record template with multiple slots. The medical record generation module is used to fill the medical key information with the intent tag into the corresponding slots of the multiple slots of the outpatient medical record template according to the intent tag, generate an outpatient medical record draft, and transmit the outpatient medical record draft to the interactive display module. The interactive display module is used to receive operation instructions for the draft outpatient medical record, generate a confirmed outpatient medical record, and transmit the confirmed outpatient medical record to the storage and interface module; The storage and interface module is configured with a data interface, which is used to store the confirmed outpatient medical records to the local computing device and transmit the confirmed outpatient medical records to an external hospital information system through the data interface.

2. The specialized medical record robot system based on a large vertical domain model according to claim 1, characterized in that, The preprocessing and noise reduction module performs time-frequency domain conversion on the dialogue speech signal to generate a noisy speech spectrum sequence. The noisy speech spectrum sequence contains multiple noisy speech spectrum frames, each with a corresponding amplitude. The multiple noisy speech spectrum frames include a target noisy speech spectrum frame. The preprocessing and noise reduction module is configured with a preset energy threshold. The preprocessing and noise reduction module performs endpoint detection on the noisy speech spectrum sequence, calculates the short-time energy of each noisy speech spectrum frame, compares each short-time energy with the preset energy threshold, marks noisy speech spectrum frames with short-time energy lower than the preset energy threshold as non-speech segments, and calculates the estimated spectral amplitude of the background noise corresponding to each noisy speech spectrum frame based on the non-speech segments. The preprocessing and noise reduction module is configured with a preset adaptive over-subtraction factor and a preset lower spectral limit constant. The preprocessing and noise reduction module uses an adaptive spectral subtraction algorithm with a lower spectral limit to process the noisy speech spectrum sequence and calculate the spectrum estimate of the clean speech. The spectrum estimate of the clean speech is obtained by comparing a first operation value and a second operation value and taking the maximum value. The first operation value is the amplitude corresponding to the target noisy speech spectrum frame, minus the product of the preset adaptive over-subtraction factor and the spectrum amplitude estimate of the background noise corresponding to the target noisy speech spectrum frame. The second operation value is the product of the preset lower spectral limit constant and the amplitude corresponding to the target noisy speech spectrum frame.

3. The specialized medical record robot system based on a large vertical domain model according to claim 1, characterized in that, The cross-modal language processing module is configured with a sliding window, and the cross-modal language processing module has a pre-stored probability judgment threshold; the natural language processing model is configured with fine-tuning parameters; The cross-modal language processing module receives the intermediate recognized text in real time through the sliding window to extract the contextual data contained in the intermediate recognized text; Based on the fine-tuning parameters of the natural language processing model, an autoregressive prediction operation is performed on the contextual data to calculate the predicted probability of each medical subdomain label; the cross-modal language processing module extracts medical subdomain labels whose predicted probabilities exceed the probability judgment threshold as the target medical subdomain labels; the temporary hot word set contains multiple candidate medical hot words, the cross-modal language processing module calculates the prior probability of each candidate medical hot word, and inputs the temporary hot word set and the prior probabilities of each candidate medical hot word in reverse to the dynamic speech recognition module.

4. The specialized medical record robot system based on a large vertical domain model according to claim 3, characterized in that, The dynamic speech recognition module is equipped with a basic medical hot word dictionary; The dynamic speech recognition module receives the temporary hot word set and the prior probability of each candidate medical hot word, and calculates the fusion weight of each candidate medical hot word. The fusion weight is the sum of the basic static weight and the dynamic feedback adjustment term; the basic static weight is the weight assigned to the candidate medical hot words by the basic medical hot word dictionary; the dynamic feedback adjustment term is the product of a preset dynamic feedback adjustment coefficient, the prior probability of the candidate medical hot words, and a time decay factor; the time decay factor is determined based on a preset time decay constant and a time difference, and the time difference is the time difference between the current operation time and the trigger time when the cross-modal language processing module calculates the prior probability; The dynamic speech recognition module calculates the acoustic log-likelihood score corresponding to each candidate medical hot word in the decoding graph search path, injects each fusion weight into the decoding graph search path, and superimposes each fusion weight into the acoustic log-likelihood score of the corresponding candidate medical hot word. The dynamic speech recognition module updates the values ​​of each fusion weight in real time according to the time decay factor, and adjusts the node retention status of each candidate medical hot word in the decoding graph search path according to the updated fusion weight.

5. The specialized medical record robot system based on a large vertical domain model according to claim 1, characterized in that, The dialogue text sequence contains multiple word units; the acoustic sequence metadata includes acoustic confidence scores and pronunciation pause features. The dynamic speech recognition module synchronously extracts the acoustic sequence metadata corresponding to the dialogue text sequence, and combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, specifically including: The dynamic speech recognition module extracts the acoustic sequence metadata corresponding to each word in the dialogue text sequence, aligns the dialogue text sequence with the acoustic sequence metadata in the time dimension, and combines them into the multimodal data stream.

6. The specialized medical record robot system based on a large vertical domain model according to claim 5, characterized in that, The cross-modal language processing module identifies target medical entities in the dialogue text sequence through the natural language processing model, extracts the target medical entities in the dialogue text sequence to generate the key medical information, and records the key medical information as a medical entity sequence; The medical entity sequence contains the corresponding target words in the dialogue text sequence, and the medical entity sequence has a sequence position and a pronunciation preposition position. The acoustic sequence metadata has a data structure. The cross-modal language processing module aligns the medical entity sequence with the acoustic sequence metadata using multimodal features, compares the sequence position with the data structure, extracts the acoustic confidence score in the acoustic sequence metadata corresponding to each target word, and calculates the average acoustic confidence score of all target words contained in the medical entity sequence. The cross-modal language processing module extracts the pronunciation pause features in the acoustic sequence metadata corresponding to each target word, and identifies the pronunciation preposition position and the total duration of pronunciation pauses within all target words based on the pronunciation pause features.

7. The specialized medical record robot system based on a large vertical domain model according to claim 6, characterized in that, The cross-modal language processing module has a pre-stored mapping table of limited modifiers and score weights, as well as medical judgment thresholds, and is configured with a first normalized weight coefficient, a second normalized weight coefficient, and a third normalized weight coefficient. The medical entity sequence has contextual content; the cross-modal language processing module extracts the contextual content, compares the contextual content with the mapping table, and extracts the corresponding score weight as the semantic confidence score based on the mapping table; The diagnostic score calculated by the cross-modal language processing module is the difference between the sum of the acoustic weighting term and the semantic weighting term and the pause penalty term; the acoustic weighting term is the product of the average acoustic confidence score and the first normalized weighting coefficient, the pause penalty term is the product of the total duration of the pronunciation pause and the second normalized weighting coefficient, and the semantic weighting term is the product of the semantic confidence score and the third normalized weighting coefficient. The cross-modal language processing module compares the diagnostic score with the medical judgment threshold. When the diagnostic score is greater than or equal to the medical judgment threshold, it assigns a diagnostic category intent label to the medical entity sequence. When the diagnostic score is less than the medical judgment threshold, an intent label of suspected category is assigned to the medical entity sequence.

8. The specialized medical record robot system based on a large vertical domain model according to claim 1, characterized in that, The outpatient medical record template includes multiple slots such as chief complaint slot, present illness slot, past medical history slot, physical signs slot, diagnostic opinion slot, medication recommendation slot, and examination item slot; the outpatient medical record template is configured with confirmed and suspected sub-slots under the diagnostic opinion slot and the medication recommendation slot; The medical key information with the intent tag includes medical entity content and the intent tag, and the medical entity content has corresponding attributes; the medical record generation module extracts the medical entity content and the intent tag contained in the medical key information with the intent tag, matches the corresponding slot in the plurality of slots according to the corresponding attributes of the medical entity content, and further matches the confirmed sub-slot or the suspected sub-slot according to the category of the intent tag for filling.

9. The specialized medical record robot system based on a large vertical domain model according to claim 1, characterized in that, The local computing device is equipped with an external input device. The interactive display module receives the operation instructions transmitted by the external input device. The operation instructions include editing data and confirmation data for the draft outpatient medical record. The draft outpatient medical record contains draft text content. The interactive display module updates the draft text content of the draft outpatient medical record according to the operation instructions and generates the confirmed outpatient medical record. The data interface is an encrypted data transmission interface; the storage and interface module receives a data synchronization instruction, performs a data encryption operation on the confirmed outpatient medical record according to the data synchronization instruction, and transmits the encrypted confirmed outpatient medical record to the external hospital information system through the encrypted data transmission interface.

10. A specialized medical record robot method based on a large vertical domain model, characterized in that, The specialist medical record robot system based on a vertical domain large model, as described in claim 1, comprises the following steps: S1. The voice acquisition module acquires the dialogue voice signal and transmits it to the preprocessing and noise reduction module; the preprocessing and noise reduction module performs time-frequency domain conversion and noise suppression operations on the dialogue voice signal, filters out background noise and generates a clear voice signal, and transmits the clear voice signal to the dynamic voice recognition module. S2. The dynamic speech recognition module receives the clear speech signal and performs acoustic decoding. The acoustic decoding operation generates a dialogue text sequence based on the decoding graph search path, and continuously outputs intermediate recognized text to the cross-modal language processing module during the acoustic decoding operation. The cross-modal language processing module has multiple medical subdomain tags pre-stored. The intermediate recognized text contains contextual data. The cross-modal language processing module extracts the contextual data, calculates the prediction probability of each medical subdomain tag based on the contextual data, extracts the target medical subdomain tag, generates a temporary hot word set based on the target medical subdomain tag, and inputs the temporary hot word set back to the dynamic speech recognition module. The dynamic speech recognition module adjusts the decoding graph search path according to the temporary hot word set; S3. When generating the dialogue text sequence, the dynamic speech recognition module simultaneously extracts the acoustic sequence metadata corresponding to the dialogue text sequence, combines the dialogue text sequence and the acoustic sequence metadata into a multimodal data stream, and transmits it to the cross-modal language processing module. S4. The cross-modal language processing module receives the multimodal data stream and performs medical entity extraction, outputting key medical information. The key medical information includes text content. The cross-modal language processing module calculates a semantic confidence score based on the text content. The cross-modal language processing module combines the semantic confidence score with the acoustic sequence metadata to jointly evaluate the key medical information and calculate a diagnostic accuracy score. The cross-modal language processing module assigns a corresponding intent tag to the key medical information according to the diagnostic accuracy score and transmits the key medical information with the intent tag to the medical record generation module. S5. The medical record generation module has a pre-stored outpatient medical record template containing multiple slots. The medical record generation module fills the key medical information with the intent tag into the corresponding slots of the multiple slots of the outpatient medical record template according to the intent tag, generates an outpatient medical record draft, and transmits it to the interactive display module. The interactive display module receives operation instructions for the draft outpatient medical record, generates a confirmed outpatient medical record, and transmits the confirmed outpatient medical record to the storage and interface module. The storage and interface module is configured with a data interface. The storage and interface module receives the confirmed outpatient medical record and stores it in the local computing device, and transmits the confirmed outpatient medical record to an external hospital information system through the data interface.

Images