A medical care voice structured input method and system based on self-supervised learning

Abstract

The application relates to a medical care voice structured input method and system based on self-supervised learning, and belongs to the medical service field. The method comprises the following steps: collecting medical care voice, outputting an audio feature sequence and a context high-dimensional vector based on the medical care voice; pre-training an MS-CCP model through self-supervised learning, outputting a medical voice encoder through the MS-CCP model; inputting the audio feature sequence into the medical voice encoder to obtain a high-level semantic feature sequence, performing dynamic memory and reasoning decoding based on the high-level semantic feature sequence and the context high-dimensional vector, outputting a word sequence and a label sequence, and merging to form a structured voice record; and performing structured confidence verification and writing based on the structured voice record. The application realizes medical voice structured input which overcomes dependence on labeled data, fuses patient context information, realizes end-to-end structured output, and has intelligent confidence evaluation.

Description

Technical Field

[0001] This invention belongs to the field of medical service technology, specifically relating to a method and system for structured input of medical and nursing voice based on self-supervised learning. Background Technology

[0002] In the context of the rapid development of medical informatization, voice input technology has become an important tool for improving clinical work efficiency and reducing the burden on medical staff. Traditional medical records mainly rely on manual entry, which is not only time-consuming and labor-intensive, but also prone to hand contamination and errors due to fatigue and negligence, affecting medical quality and patient safety. Meanwhile, in accordance with the spirit of the National Health Commission's document on simplifying medical record writing, it is necessary to reduce unnecessary and repetitive paperwork, allowing nurses more time to engage in clinical work, optimize and strengthen the recording requirements of core aspects related to medical quality and safety, and emphasize the accuracy, timeliness, logic, and intrinsic quality of records so that they truly reflect the entire diagnosis and treatment process and clinical thinking. Therefore, the development of efficient, accurate, and intelligent medical voice input systems has significant clinical necessity and urgency. Existing medical voice input technologies mainly include rule-based template methods, statistical acoustic models, and end-to-end deep learning speech recognition systems. While these methods have achieved some success in general domains, they still have significant shortcomings in the highly specialized and complex context of medicine: First, medical terminology is numerous, pronunciations vary greatly, and English and Chinese are often mixed, resulting in a lack of domain adaptability in general models and a high error rate. Second, most systems only convert speech to text, lacking an understanding of semantic structure and unable to output structured data that can be directly embedded in electronic medical records. Third, models heavily rely on large amounts of labeled corpora for supervised training, but medical speech data is difficult to obtain due to privacy concerns, high labeling thresholds, and significant dialect and professional variations, thus limiting model performance. Fourth, existing systems generally lack the ability to deeply integrate contextual information and cannot combine individual patient history (such as medical history, medication, and test results) for real-time reasoning and error correction, leading to a significant increase in error rates in cases of accent, noise, or semantic ambiguity. Fifth, the confidence assessment mechanism for recognition results is weak, and the system cannot effectively warn or perform secondary verification for low-confidence outputs, potentially posing clinical risks. Therefore, developing a medical voice structured recording method that can overcome dependence on labeled data, integrate patient context information, achieve end-to-end structured output, and possess intelligent confidence assessment has become one of the key technological challenges driving the development of smart healthcare. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this invention provides a method and system for structured recording of medical and nursing voice based on self-supervised learning.

[0004] The objective of this invention can be achieved through the following technical solutions: A structured voice recording method for medical staff based on self-supervised learning, the implementation of which includes the following steps: Step S1: Collect medical staff's voice recordings, and output audio feature sequences and high-dimensional context vectors based on the medical staff's voice recordings; Step S2: Pre-train the MS-CCP model through self-supervised learning, and produce a medical speech encoder through the MS-CCP model; Step S3: Input the audio feature sequence into the medical speech encoder to obtain the high-level semantic feature sequence. Perform dynamic memory reasoning decoding based on the high-level semantic feature sequence and the high-dimensional context vector to output the word sequence and tag sequence, and merge them to form a structured speech record. Step S4: Perform structured confidence verification and writing based on the structured speech recording.

[0005] Preferably, step S1 specifically includes: The medical staff's voice recordings are collected in real time and subjected to noise reduction and active voice detection. The audio feature sequence is obtained by calculating the Mel spectrogram. Based on the patient ID identified by the medical staff's voice, the high-dimensional context vector is retrieved from the database in real time based on the patient ID. The high-dimensional context vector is obtained by mapping the original data in the database through an embedding layer and includes static and dynamic information.

[0006] Preferably, the pre-training of the MS-CCP model in step S2 specifically includes: Collect medical scenario voice data to form a medical scenario voice dataset; An audio segment is randomly selected from the medical scene speech dataset as a mask segment. The mask segment is replaced with a silent mask. The medical scene speech dataset after the mask segment is replaced is used as training audio input into the Student Encoder, which is a Transformer encoder. The original medical scene speech dataset is input into the Teacher Encoder. The Teacher Encoder has the same structure as the Student Encoder, but the parameter update method is different. The parameters of the Teacher Encoder are... By the parameters of Student Encoder The exponential moving average is obtained; The masking segment is divided into masking points. For each masking point, the Student Encoder outputs a masking feature vector, and the Teacher Encoder outputs a template feature vector at the corresponding position of the masking point. The MS-CCP loss function is constructed based on the masking feature vector and the template feature vector, mathematically described as follows: ,in, For MS-CCP loss function, For cover MS-CCP loss function, For cover The cosine similarity between the masked feature vector and the corresponding template feature vector. For cover The masking feature vector, For cover The corresponding template feature vector, For the remaining template feature vectors, For cover The cosine similarity between the masked feature vector and the remaining template feature vectors. For temperature hyperparameters, ≠ ; The Student Encoder is trained using the MS-CCP loss function to produce the medical speech encoder.

[0007] Preferably, the dynamic memory reasoning decoding in step S3 specifically includes: A memory matrix is ​​constructed based on the high-dimensional context vector, and the memory matrix contains multiple knowledge fragments; The decoder begins decoding, obtains the current hidden state of the decoder, and uses an attention mechanism to look at the high-level semantic feature sequence to obtain the sequence of key attention; The current hidden state of the decoder and the key attention sequence are combined to form a query. The similarity score is calculated between the query and all the knowledge fragments in the memory matrix. The attention weight of each knowledge fragment is obtained based on the similarity score. The attention weights are used to perform a weighted summation on the corresponding rows of the memory matrix and output a memory vector. The memory vector and the high-level semantic feature sequence are concatenated as the key and value in the attention mechanism, and the word sequence and the label sequence are output.

[0008] Preferably, step S4 specifically includes: A capsule network is introduced to convert the structured speech recording into an input vector and input it into the capsule network. The core mechanism of the capsule network is a dynamic routing mechanism. Obtain the output vector of the capsule network and input it into the reconstruction network to obtain the reconstruction vector; The confidence score is obtained based on the input vector and the reconstructed vector, mathematically described as follows: Where S is the confidence score and a is the capsule activation value. Here, u is the scaling hyperparameter, v is the input vector, and v is the reconstructed vector. This represents the reconstruction error; Structured confidence verification and writing are performed based on the confidence score.

[0009] A structured voice recording system for medical and nursing staff based on self-supervised learning is used to execute the structured voice recording method for medical and nursing staff described above, including an audio processing module, an encoder training module, a structuring module, and a confidence verification module. The audio processing module is used to collect medical staff's voice and output audio feature sequences and high-dimensional context vectors based on the medical staff's voice. The encoder training module is used to pre-train the MS-CCP model through self-supervised learning, and to produce a medical speech encoder through the MS-CCP model. The structuring module is used to input the audio feature sequence into the medical speech encoder to obtain a high-level semantic feature sequence, perform dynamic memory reasoning decoding based on the high-level semantic feature sequence and the high-dimensional context vector, output a word sequence and a tag sequence, and merge them to form a structured speech record; The high-dimensional context vector is used for structured confidence verification and writing based on the structured speech recording.

[0010] The beneficial effects of this invention are as follows: (1) By introducing a self-supervised learning mechanism, the dependence on labeled data is significantly reduced, and rich semantic representations can be learned from massive unlabeled medical speech, thereby improving the model’s generalization ability under changes in professional terminology and context.

[0011] (2) By retrieving and fusing the patient’s high-dimensional contextual information in real time, it has stronger semantic perception and correction capabilities during the decoding process.

[0012] (3) With the help of the dynamic routing and reconstruction mechanism of capsule network, the output results can be intelligently evaluated to effectively distinguish between highly credible and suspicious outputs.

[0013] (4) The final output is a semantic unit with a clear label structure rather than ordinary text, which can be directly integrated with the hospital information system. This not only greatly reduces the paperwork burden of medical staff and improves work efficiency, but also improves the quality and usability of medical data through structured and verifiable data entry. Attached Figure Description

[0014] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0015] Figure 1 This is a flowchart illustrating the steps of a structured voice recording method for medical and nursing staff based on self-supervised learning, as described in this invention. Detailed Implementation

[0016] To better understand the invention, various aspects of the invention will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of the invention and are not intended to limit the scope of the invention in any way. Throughout the specification, the expression "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the terms "approximately," "about," and similar terms are used as expressions of approximation, not as expressions of degree, and are intended to describe inherent deviations in measured or calculated values ​​that will be recognized by those skilled in the art. Furthermore, the order in which the steps are described in this invention does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0017] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of the invention, the word "may" is used to mean "one or more embodiments of the invention." And the term "exemplary" is intended to refer to examples or illustrations.

[0018] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formalized sense.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Example 1: Please see Figure 1 A structured voice recording method for medical staff based on self-supervised learning includes: S1: Collect medical staff's voice recordings, and output audio feature sequences and high-dimensional context vectors based on the medical staff's voice recordings; S2: Pre-train the MS-CCP model (masked speech-context contrast prediction model) through self-supervised learning, and produce a medical speech encoder through the MS-CCP model; S3: Input the audio feature sequence into the medical speech encoder to obtain the high-level semantic feature sequence, perform dynamic memory reasoning decoding based on the high-level semantic feature sequence and the high-dimensional context vector, output the word sequence and the tag sequence, and merge them to form a structured speech record; S4: Perform structured confidence verification and writing based on the structured speech recording.

[0021] In this embodiment, based on the audio feature sequence of the medical staff's voice output and the high-dimensional context vector, it can be implemented through the following steps: S101: During clinical rounds, the medical staff's voice is collected in real time using devices such as microphones worn by the medical staff, and noise reduction and active voice detection are performed (silent segments are removed, and effective voice segments are retained). S102: Feature extraction is performed by calculating the Mel spectrogram (a two-dimensional matrix, with time on the horizontal axis and frequency bins on the vertical axis, where the value of each point represents energy and is a normalized representation of the audio) to obtain the audio feature sequence (a T×F matrix, where T is the time step and F is the number of Mel bands). S103: Based on the patient ID (bed number, social security card number, etc.) recognized by the medical staff voice, the context high-dimensional vector of the patient is retrieved in real time from the database (such as the hospital information system) based on the patient ID. The context high-dimensional vector is obtained by mapping the original data in the database through the embedding layer, including static information (age, gender, past medical history, etc.) and dynamic information (most recent vital signs record, current medication, etc.).

[0022] In this embodiment, the pre-training of the MS-CCP model can be implemented through the following steps: S201: Collect massive amounts of unlabeled medical scenario speech data to form a medical scenario speech dataset (e.g., a large number of recordings of medical staff making rounds). S202: Randomly select 15% of the audio segments in the medical scene speech dataset as mask segments, replace the mask segments with a silent mask or random noise, and input the medical scene speech dataset after replacing the mask segments as training audio into the Student Encoder, where the Student Encoder is a Transformer encoder. S203: Input the original medical scene speech dataset into the Teacher Encoder. The Teacher Encoder has the same structure as the Student Encoder, but the parameter update method is different. The parameters of the Teacher Encoder... It is not updated directly through gradient descent, but rather by the parameters of the Student Encoder. The exponential moving average is obtained, mathematically described as follows: ,in, For the new Teacher Encoder parameters, The parameters for Teacher Encoder from the previous step. The momentum coefficient is a value close to 1 (e.g., 0.99), which means that with each update, the Teacher Encoder only absorbs a small amount of new knowledge from the Student Encoder, while retaining the majority of its original state. This ensures that the Teacher Encoder is more stable and smoother, and will not be swayed by a single error in the Student Encoder, thus providing a more reliable learning target. S204: Divide the masking segment into masking points. For each masking point, output a masking feature vector through the StudentEncoder, and at the same time output a template feature vector at the position corresponding to the masking point through the TeacherEncoder (that is, find the position of the masking point in the original medical scene speech dataset, and output the feature vector at that position through the TeacherEncoder). S205: Construct the MS-CCP loss function based on the masking feature vector and the template feature vector, mathematically described as follows: ,in, For MS-CCP loss function, For cover MS-CCP loss function, For cover The cosine similarity between the masked feature vector and the corresponding template feature vector. For cover The masking feature vector, For cover The corresponding template feature vector, For the remaining template feature vectors, For cover The cosine similarity between the masked feature vector and the remaining template feature vectors. For temperature hyperparameters, ≠ The MS-CCP loss function is used to adjust the sharpness of the distribution. Its purpose is to make the similarity between the masked feature vector and the template feature vector at the corresponding position much greater than the similarity between the masked feature vector and the template feature vector at other positions. Once the above purpose is achieved, the value of the MS-CCP loss function will naturally decrease, and vice versa. Through the above steps, the Student Encoder will be forced to learn a key ability: instead of memorizing the specific pronunciation of a word, it will learn the semantics and contextual relationships of that word in medical dialogues. For example, after learning, the Student Encoder will find that the feature vectors of words like "body temperature" and "38.5" and "Celsius" are very close in space, while the feature vectors of words like "blood pressure" and "120" and "mmHg" are very close in another region.

[0023] S206: Train the Student Encoder using the MS-CCP loss function and produce the medical speech encoder.

[0024] In this embodiment, the dynamic memory reasoning decoding specifically refers to: S301: Construct a memory matrix based on the high-dimensional context vector. The memory matrix contains multiple knowledge fragments (each row represents a knowledge fragment, such as M[1]=[Diagnosis: Hypertension]; M[2]=[Last Blood Pressure: 160 / 100 mmHg]; M[3]=[Age: 72]; M[4]=[Gender: Male]). S302: The decoder starts decoding, obtains the current hidden state of the decoder, and uses an attention mechanism to look at the high-level semantic feature sequence to obtain the key attention sequence; S303: Combine the current hidden state of the decoder and the key attention sequence to form a query. Calculate the similarity score between the query and all the knowledge fragments in the memory matrix. Obtain the attention weight of each knowledge fragment based on the similarity score (the higher the similarity score, the greater the attention weight, representing which knowledge fragments are most relevant to the current state). S304: Use the attention weights to perform a weighted summation on the corresponding rows of the memory matrix and output the memory vector; S305: Concatenate the memory vector and the high-level semantic feature sequence as the Key and Value in the attention mechanism (the core of the decoder is the multi-head attention mechanism; traditional attention usually only focuses on the encoder's output, i.e., the high-level semantic feature sequence), and output the parallel word sequence and the label sequence (the label sequence only requires adding an extra parallel task to label each generated word).

[0025] Example: The input audio is "Patient in bed 25, temperature 38 degrees Celsius, blood pressure 150 / 95", the encoder converts this sentence into a high-level semantic feature sequence: The decoder begins decoding, outputting words one by one. When it outputs the word "body temperature" and prepares to generate the next word, the decoder's current hidden state already contains "body temperature". <start>The historical information "Patient in Bed No. 25 with body temperature" is then analyzed using an attention mechanism to examine the high-level semantic feature sequence output by the encoder, identifying the most important sequence to focus on. The current hidden state and The query is composed of elements and it is found that the similarity between the element and each row in the memory matrix M={M[1],M[2],M[3],M[4]} is very low. This means that there is no relevant historical record of body temperature in the patient's information. This means that the attention weight of all rows is very low. Therefore, the memory vector read is a "vague summary" with little information. The information after fusion is very little. This is equivalent to telling the decoder that there is no special body temperature information in the patient's history that needs special attention. The decoder can trust the sound it hears and its own judgment. Then the decoder outputs the next word "38" with a very high confidence. When the word "blood pressure" is output, the current hidden state of the decoder has already contained all the previous information. The attention mechanism looks at the next word. The features, after querying, are found to be very similar to M[1] and M[2] in the memory matrix. The attention weights of these two segments are extremely large. The memory vector formed is then passed to the decoder with a strong signal, that is, the patient's blood pressure is usually very high and special attention needs to be paid to the information heard next. When the decoder decodes the "150" word, based on the information previously obtained, it is found that the value being decoded is indeed the high blood pressure value, with a high confidence level, and can be directly output (if the decoder finds that the decoded word seems to have a problem, such as decoding the "20" word after the "blood pressure" word, it will issue the corresponding warning information). In this way, two parallel sequences are finally generated: word sequence: [ <start>Patient in bed number 25, body temperature 38 degrees Celsius, blood pressure 150 / 95; tag sequence: [O,O,O,O,O,VITAL,VALUE,UNIT,VITAL,VALUE,O,VALUE]; merging these two parallel sequences forms the structured speech record {"body temperature": 38, "systolic blood pressure": 150, "diastolic blood pressure": 95}.

[0026] In this embodiment, structured confidence verification and writing based on the structured speech recording can be implemented through the following steps: S401: Introduce a capsule network to convert the structured speech recording into an input vector and input it into the capsule network. The core mechanism of the capsule network is a dynamic routing mechanism. The dynamic routing mechanism is as follows: The capsule network consists of two layers. The first layer is a primary capsule, where each capsule outputs a preliminary prediction vector for the input vector. The second layer is a digital capsule. The number of capsules corresponds to the category to be evaluated. It can be simply set to two capsules, capsule T and capsule F. Capsule T represents a reliable value, and capsule F represents an unreliable value. After the input vector is input, each primary capsule predicts which number capsule it should belong to and projects its output onto the predicted number capsule. After the input is complete, the two number capsules collect all the received outputs and sum them to obtain two original output vectors. Compress the two original output vectors separately to obtain the final output vector (V). T and V F The compression operation preserves the direction of the vector, but compresses its length to between 0 and 1. The length of the compressed vector is used as the capsule activation value of the corresponding capsule (the highest capsule activation value is ultimately selected as the basis for subsequent confidence analysis). The above process is iteratively updated by checking the consistency between the prediction of each primary capsule and the final output. If the dot product of the vector output by the primary capsule and the vector output by the digital capsule is large, the coupling coefficient of the primary capsule in the next iteration is increased; otherwise, it is decreased. This process is repeated 3-5 times.

[0027] S402: Obtain the output vector (V) of the capsule network. T and V F The reconstructed vector is then input into the reconstruction network (usually a simple fully connected decoder) to obtain the reconstructed vector. S403: Based on the input vector and the reconstructed vector, a confidence score is obtained, mathematically described as follows: Where S is the confidence score and a is the capsule activation value. Here, u is the scaling hyperparameter, v is the input vector, and v is the reconstructed vector. This represents the reconstruction error; S404: Perform structured confidence verification and writing based on the confidence score. Example: Preset high and low thresholds. Records exceeding the high threshold are assigned high confidence and directly written to the electronic medical record system. Records below the low threshold are assigned low confidence and prompt medical staff to rewrite. Records with medium confidence are intelligently confirmed, such as by asking for confirmation, where medical staff only need to answer yes or no.

[0028] Example 2: A structured voice recording system for medical and nursing staff based on self-supervised learning includes an audio processing module, an encoder training module, a structuring module, and a confidence verification module. The audio processing module is used to collect medical staff's voice and output audio feature sequences and high-dimensional context vectors based on the medical staff's voice. The encoder training module is used to pre-train the MS-CCP model through self-supervised learning, and to produce a medical speech encoder through the MS-CCP model. The structuring module is used to input the audio feature sequence into the medical speech encoder to obtain a high-level semantic feature sequence, perform dynamic memory reasoning decoding based on the high-level semantic feature sequence and the high-dimensional context vector, output a word sequence and a tag sequence, and merge them to form a structured speech record; The high-dimensional context vector is used for structured confidence verification and writing based on the structured speech recording.

[0029] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.< / start> < / start>

Claims

1. A method for structured recording of medical and nursing voice based on self-supervised learning, characterized in that, The implementation of the structured voice recording method for medical staff includes the following steps: Step S1: Collect medical staff's voice recordings, and output audio feature sequences and high-dimensional context vectors based on the medical staff's voice recordings; Step S2: Pre-train the MS-CCP model through self-supervised learning, and produce a medical speech encoder through the MS-CCP model. The MS-CCP model is a masked speech-context contrast prediction model. Step S3: Input the audio feature sequence into the medical speech encoder to obtain the high-level semantic feature sequence. Perform dynamic memory reasoning decoding based on the high-level semantic feature sequence and the high-dimensional context vector to output the word sequence and tag sequence, and merge them to form a structured speech record. Step S4: Perform structured confidence verification and writing based on the structured speech recording.

2. The medical and nursing voice structured recording method according to claim 1, characterized in that, Step S1 specifically includes: The medical staff's voice recordings are collected in real time and subjected to noise reduction and active voice detection. The audio feature sequence is obtained by calculating the Mel spectrogram. Based on the patient ID identified by the medical staff's voice, the high-dimensional context vector is retrieved from the database in real time based on the patient ID. The high-dimensional context vector is obtained by mapping the original data in the database through an embedding layer and includes static and dynamic information.

3. The medical and nursing voice structured recording method according to claim 1, characterized in that, The pre-training of the MS-CCP model in step S2 specifically includes: Collect medical scenario voice data to form a medical scenario voice dataset; An audio segment is randomly selected from the medical scene speech dataset as a mask segment. The mask segment is replaced with a silent mask. The medical scene speech dataset after the mask segment is replaced is used as training audio input into the Student Encoder, which is a Transformer encoder. The original medical scene speech dataset is input into the Teacher Encoder. The Teacher Encoder has the same structure as the Student Encoder, but the parameter update method is different. The parameters of the Teacher Encoder are... By the parameters of Student Encoder The exponential moving average is obtained; The masking segment is divided into masking points. For each masking point, the Student Encoder outputs a masking feature vector, and the Teacher Encoder outputs a template feature vector at the corresponding position of the masking point. The MS-CCP loss function is constructed based on the masking feature vector and the template feature vector, mathematically described as follows: ,in, For MS-CCP loss function, Let i be the MS-CCP loss function for the masking point. Let be the cosine similarity between the masking feature vector of masking point i and the corresponding template feature vector. Let i be the masking feature vector of the masking point. Let i be the template feature vector corresponding to the masking point. For the remaining template feature vectors, Let be the cosine similarity between the masking feature vector of masking point i and the feature vectors of the remaining templates. Temperature hyperparameter, ≠ ; The Student Encoder is trained using the MS-CCP loss function to produce the medical speech encoder.

4. The medical and nursing voice structured recording method according to claim 1, characterized in that, The dynamic memory reasoning decoding in step S3 specifically refers to: A memory matrix is ​​constructed based on the high-dimensional context vector, and the memory matrix contains multiple knowledge fragments; The decoder begins decoding, obtains the current hidden state of the decoder, and uses an attention mechanism to look at the high-level semantic feature sequence to obtain the sequence of key attention; The current hidden state of the decoder and the key attention sequence are combined to form a query. The similarity score is calculated between the query and all the knowledge fragments in the memory matrix. The attention weight of each knowledge fragment is obtained based on the similarity score. The attention weights are used to perform a weighted summation on the corresponding rows of the memory matrix and output a memory vector. The memory vector and the high-level semantic feature sequence are concatenated as the key and value in the attention mechanism, and the word sequence and the label sequence are output.

5. The medical and nursing voice structured recording method according to claim 1, characterized in that, Step S4 specifically includes: A capsule network is introduced to convert the structured speech recording into an input vector and input it into the capsule network. The core mechanism of the capsule network is a dynamic routing mechanism. Obtain the output vector of the capsule network and input it into the reconstruction network to obtain the reconstruction vector; The confidence score is obtained based on the input vector and the reconstructed vector, mathematically described as follows: Where S is the confidence score and a is the capsule activation value. Here, u is the scaling hyperparameter, v is the input vector, and v is the reconstructed vector. This represents the reconstruction error; Structured confidence verification and writing are performed based on the confidence score.

6. A structured voice input system for medical and nursing staff based on self-supervised learning, characterized in that, The system is applied to the medical and nursing voice structured recording method as described in any one of claims 1-5, and includes an audio processing module, an encoder training module, a structured module, and a confidence verification module; The audio processing module is used to collect medical staff's voice and output audio feature sequences and high-dimensional context vectors based on the medical staff's voice. The encoder training module is used to pre-train the MS-CCP model through self-supervised learning, and to produce a medical speech encoder through the MS-CCP model. The MS-CCP model is a masked speech-context contrast prediction model. The structuring module is used to input the audio feature sequence into the medical speech encoder to obtain a high-level semantic feature sequence, perform dynamic memory reasoning decoding based on the high-level semantic feature sequence and the high-dimensional context vector, output a word sequence and a tag sequence, and merge them to form a structured speech record; The high-dimensional context vector is used for structured confidence verification and writing based on the structured speech recording.

Images