Intelligent interactive self-service machine and system based on speech recognition

Abstract

The application relates to the technical fields of intelligent voice interaction and medical informatization, and discloses an intelligent interaction self-service machine and system based on voice recognition, wherein the intelligent interaction self-service system based on voice recognition comprises an audio acquisition module, a voice recognition module, a user identity recognition module, a semantic understanding module, an intelligent reasoning module and a dialogue management module; the audio acquisition module acquires multi-channel audio signals, extracts voice signals of a target user and carries out signal processing; the voice recognition module carries out voice text recognition and confidence evaluation; the user identity recognition module recognizes the user identity; the semantic understanding module carries out semantic understanding and intent recognition; the intelligent reasoning module carries out intelligent reasoning and decision-making, recommends a department and formulates a treatment plan; the dialogue management module executes multi-round dialogue management and information completion; the business execution module carries out privacy protection processing, and carries out task distribution and execution; the application adopts a cloud-edge-end collaborative architecture and privacy protection technology, and realizes efficient business processing under the premise of guaranteeing data security.

Description

Technical Field

[0001] This invention relates to the fields of intelligent voice interaction and medical information technology, and more specifically, to an intelligent interactive self-service machine and system based on voice recognition. Background Technology

[0002] With the deepening of medical informatization and the continuous growth of patients' medical needs, major hospitals have generally deployed self-service registration machines and self-service payment machines to alleviate the pressure on manual services and improve service efficiency. However, existing self-service equipment mainly relies on touch screen operation, which has many limitations. Touch screen operation is extremely unfriendly to the elderly, visually impaired people, and patients whose hands are occupied. The operation is complex and has a high learning cost, causing these users to still rely on manual services, failing to truly achieve service diversion. Secondly, in the context of normalized disease prevention and control, frequent touch operations increase the risk of cross-infection, and patients have psychological concerns about contact-based operations.

[0003] While some voice-interactive self-service devices have emerged in the market, they face significant challenges in the unique environment of hospitals. Hospitals are characterized by high population density and complex, variable background noise, including conversations, medical equipment operation, and announcements. Traditional speech recognition technology experiences a significant drop in accuracy under these conditions. Furthermore, hospital users span a wide age range, speak diverse dialects, and exhibit significant differences in voice characteristics, making it difficult for existing systems to adapt to such complex user voice features. More importantly, medical procedures are highly specialized, involving numerous medical terms, department names, and examination procedures, while users often use colloquial descriptions of symptoms. Existing speech recognition and semantic understanding technologies struggle to accurately interpret users' true intentions, leading to frequent misidentifications and erroneous operations.

[0004] Furthermore, medical service processes are complex, often requiring multiple rounds of interaction to complete a single transaction. For example, appointment booking requires confirming multiple factors such as department, doctor, and time. Existing systems lack effective multi-round dialogue management mechanisms, making it unable to handle complex user needs and changing intentions, resulting in a poor user experience. At the same time, medical data involves patient privacy, placing higher demands on data security and privacy protection. Existing systems have relatively weak technical measures for privacy protection. Summary of the Invention

[0005] This invention provides an intelligent interactive self-service machine and system based on speech recognition, which solves the technical problems of low speech recognition accuracy and poor interactive experience of traditional hospital self-service equipment in complex environments.

[0006] This invention provides an intelligent interactive self-service system based on speech recognition, comprising: The audio acquisition module is used to acquire multi-channel audio signals, extract the speech signals of the target user, and perform signal processing to obtain a speech segment sequence. The speech recognition module performs speech-to-text recognition and confidence assessment based on speech segment sequences to obtain the recognized text sequence. The user identification module identifies users based on speech segment sequences and recognized text sequences, obtaining user identification identifiers; and constructs personalized profiles based on user identification identifiers to obtain user configuration parameters. The semantic understanding module performs semantic understanding and intent recognition based on the identified text sequence and user identity identifier to obtain a structured user intent representation; The intelligent reasoning module performs intelligent reasoning and decision-making based on structured user intent representation and user identity identifiers, recommends departments and formulates treatment plans to obtain a complete treatment plan; The dialogue management module, based on structured user intent representation and complete medical treatment plan, performs multi-round dialogue management and information completion to obtain complete business execution instructions; The business execution module performs privacy protection processing based on complete business execution instructions, and allocates and executes tasks to obtain business execution results and multimodal user feedback.

[0007] In a preferred embodiment, the acquisition of multi-channel audio signals includes: Based on a circular array of six omnidirectional electret condenser microphones evenly distributed on the front frame of the self-service machine, multi-channel digital acquisition is performed to obtain the raw audio data stream. Based on the original audio data stream, the generalized cross-correlation time delay estimation algorithm is used to calculate the arrival time difference of the target speech signal received by each microphone channel, and to obtain the estimated spatial azimuth and distance of the target user relative to the center of the microphone array. Based on spatial azimuth and distance estimates, an adaptive minimum variance distortion-free response beamforming algorithm is used to perform spatial filtering on multi-channel audio signals to obtain directional speech signals.

[0008] In a preferred embodiment, the step of obtaining the speech segment sequence includes: Based on directional speech signals, an unsupervised online acoustic scene clustering algorithm is used to perform feature analysis on the current environment, and obtain acoustic scene category identifiers and scene feature description vectors. Based on acoustic scene category identifiers and scene feature description vectors, noise suppression processing is performed on directional speech signals to obtain clean speech signals for the target user. Based on the clean speech signal of the target user, an endpoint detection algorithm based on energy and spectral features is used to identify the start and end times of valid speech, and obtain a speech segment sequence.

[0009] In a preferred embodiment, the speech recognition includes: Based on the speech segment sequence, short-time Fourier transform and Mel filter bank processing are used to obtain the Mel spectral feature sequence that reflects the time-frequency characteristics of speech. Based on the Mel spectral feature sequence, a convolutional neural network is used for local feature extraction and spectral pattern recognition to obtain a high-level acoustic representation. Based on high-level acoustic representation, a multi-layer bidirectional long short-term memory network is used for temporal modeling and contextual information aggregation to obtain a hidden state sequence that integrates the preceding and following context. Based on the hidden state sequence, an encoder-decoder architecture based on an attention mechanism is used to perform sequence-to-sequence transformation to obtain the probability distribution of candidate text character sequences; Based on the candidate text character sequence, a language model that integrates medical business vocabulary and user expression habits is used for re-scoring to obtain the optimal recognized text result.

[0010] In a preferred embodiment, the confidence assessment includes calculating the confidence score of each recognized character based on the attention weight distribution and probability distribution entropy value during the decoding process using a multi-dimensional confidence assessment algorithm to obtain a recognized text sequence with character-level confidence annotations.

[0011] In a preferred embodiment, the step of obtaining the user identity identifier includes: Based on speech segment sequences, a deep speaker embedding network is used to extract the user's voiceprint features, resulting in a fixed-dimensional voiceprint feature vector. Based on the voiceprint feature vector, the cosine similarity measurement method is used to perform voiceprint matching in the local user database to obtain user identity identifiers or new user tags. Based on user identification, the system detects user registration status. If the user is already registered, it retrieves user profile data from the database to obtain user attributes and service preferences. If the user is a new user, it initiates a rapid profile building process.

[0012] In a preferred embodiment, the rapid profile building process includes: For new users, based on the acoustic features of speech segment sequences, a speech attribute classification model is used to infer user attributes, and the estimated age group, gender, and emotional state are obtained. For new users, based on the language features of the identified text sequence, dialect recognition and language style analysis algorithms are used to obtain the user's dialect type and expression habits. Based on user attribute inference results and dialect type, a transfer learning method is used to select an initialization model from a model library of similar user groups to obtain the basic service configuration for new users.

[0013] In a preferred embodiment, the step of obtaining the structured user intent representation includes: Based on the recognition of text sequences and word-level confidence annotations, a confidence-weighted text representation method is used to encode the text, resulting in a text vector representation that incorporates confidence information. Based on text vector representation, a deep neural network under a multi-task learning framework is used to perform intent classification and slot recognition, resulting in intent category labels and slot label sequences. Based on user identity and dialogue history, a dialogue state tracking model is used to maintain the context state of multi-turn dialogues and obtain a complete semantic representation.

[0014] In a preferred embodiment, the step of obtaining a complete medical plan includes: Based on the symptom and disease-related slots in the structured user intent representation, a fuzzy entity linking algorithm is used to map colloquial expressions to standard entity nodes in the medical knowledge graph, resulting in a candidate medical entity set. Based on the candidate medical entity set and user configuration parameters, a graph attention network is used to perform multi-hop relationship reasoning on the knowledge graph to obtain the probability distribution of related diseases and the recommended department list. Based on the recommended department list, real-time resource snapshots, and user's medical treatment constraints, a multi-objective constraint optimization algorithm is used to generate a personalized medical treatment path planning scheme, resulting in a complete medical treatment plan that includes department selection, time arrangement, examination items, and location navigation.

[0015] In a preferred embodiment, a voice recognition-based intelligent interactive self-service machine is provided for storing computer-readable instructions, which, when read by a computer, can operate an intelligent interactive self-service system based on voice recognition as described above.

[0016] The beneficial effects of this invention are as follows: This invention effectively solves the technical challenges of traditional hospital self-service equipment in complex environments by employing advanced technologies such as microphone arrays, adaptive beamforming, deep learning speech recognition, and multi-turn dialogue management. Through acoustic scene self-learning and adaptive noise reduction technology, the system can accurately extract the target user's voice in noisy hospital environments, improving speech recognition accuracy. Through voiceprint recognition and user profiling technology, it achieves personalized model adaptation, effectively addressing the differences in voice characteristics among users of different ages and dialect backgrounds. Through medical knowledge graphs and intelligent reasoning technology, it accurately understands users' colloquial symptom descriptions, providing professional department recommendations and consultation suggestions. Through multi-turn dialogue management and slot filling technology, it supports the complete processing of complex business processes, significantly enhancing the user interaction experience.

[0017] This invention employs a cloud-edge-device collaborative architecture and privacy protection technology to achieve efficient business processing while ensuring data security. The system supports full voice interaction, allowing users to complete the entire process of registration, inquiry, and payment without touch operation. It is particularly suitable for the elderly, visually impaired individuals, and those in disease prevention and control scenarios, effectively reducing the risk of cross-infection and improving the accessibility and inclusiveness of medical services. Through intelligent triage and personalized recommendations, the system can also optimize patient access paths, improve the efficiency of medical resource utilization, and reduce the workload of medical staff, providing crucial technical support for building smart hospitals and improving the quality of medical services. Attached Figure Description

[0018] Figure 1 This is a block diagram of an intelligent interactive self-service system based on speech recognition in this invention; Figure 2 This is a detailed flowchart of an intelligent interactive self-service system based on voice recognition, as described in this invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses an intelligent interactive self-service system based on speech recognition, such as... Figures 1 to 2 As shown, it includes: The audio acquisition module is used to acquire multi-channel audio signals, extract the speech signals of the target user, and perform signal processing to obtain a speech segment sequence. Step 1.1, Multi-channel audio signal acquisition; Based on a circular array of six omnidirectional electret condenser microphones evenly distributed on the front frame of the self-service machine, synchronous sampling technology is used to digitally acquire the ambient sound field at a sampling rate of 16 kHz, resulting in six parallel raw audio data streams. Each data stream contains timestamps and channel identification information.

[0021] The microphone array is geometrically arranged as follows: six microphone units are evenly distributed around the circumference of the front display screen of the self-service machine, with a circumference radius of 150 mm and a central angle of 60 degrees between each microphone. The center of the array is approximately 1400 mm above the ground, close to the mouth height of an adult standing. This layout design balances optimal reception sensitivity for users directly in front and omnidirectional reception capability for users to the sides and at a distance. The sampling system employs a 24-bit analog-to-digital converter and a low-noise preamplifier circuit to ensure high-fidelity signal acquisition and low noise floor.

[0022] Step 1.2, target user spatial location; Based on the original audio data stream output in step 1.1, the arrival time difference of the target speech signal received by each microphone channel is calculated using the generalized cross-correlation time delay estimation algorithm, so as to obtain the estimated spatial azimuth and distance of the target user relative to the center of the microphone array.

[0023] The core idea of ​​generalized cross-correlation time delay estimation is to utilize the time delay characteristic of signals from the same sound source received by different microphones, and find the time delay value that maximizes the correlation by calculating the cross-correlation function between channels. The specific implementation process is as follows: Perform a short-time Fourier transform on each channel signal to obtain its frequency domain representation; calculate the cross-power spectrum of the frequency domain representations of the i-th and j-th channel signals, and weight them using a phase transform weighting function, which can suppress the influence of reverberation and noise on the time delay estimation; perform an inverse Fourier transform on the weighted cross-power spectrum to obtain the generalized cross-correlation function; search for the peak position of the generalized cross-correlation function; the time corresponding to this peak is the time delay between the two channels.

[0024] By estimating the time delay of multiple microphone pairs and combining this with the geometric configuration information of the microphone array, the spatial coordinates of the sound source, including azimuth and distance information, are calculated using the principle of triangulation. The azimuth is defined as the horizontal angle between the sound source and the front of the array, ranging from -180 degrees to +180 degrees; the distance is the straight-line distance from the sound source to the center of the array, calculated using the time delay differences and sound velocity information of different microphone pairs.

[0025] Step 1.3, Adaptive beamforming processing; Based on the spatial azimuth and distance estimates output in step 1.2, the multi-channel audio signal output in step 1.1 is spatially filtered using the adaptive minimum variance distortion-free response beamforming algorithm to obtain the directional speech signal.

[0026] Beamforming enhances signals from a specific direction while suppressing interference from other directions. A minimum variance distortion-free response beamformer optimizes the weighting coefficients of each channel signal to minimize the output signal power while maintaining distortion-free transmission of the signal from the target direction, thus maximizing noise and interference suppression. Specifically, it works as follows: based on the spatial location of the target sound source and the geometric configuration of the microphone array, the theoretical phase difference of the target signal received by each microphone is calculated, and a steering vector is constructed; the covariance matrix of the input signal is estimated, reflecting the correlation and power distribution between the channel signals; by solving a constraint optimization problem, the optimal weight vector is calculated, which minimizes the output power under the unity-gain constraint in the target direction; each channel signal is multiplied by its corresponding weighting coefficient and summed to obtain the beamformed output signal. This output signal is characterized by the preservation and enhancement of the speech signal from the target direction, while noise and interference from other directions are significantly suppressed, forming a virtual beam pointing towards the target user.

[0027] Step 1.4, Acoustic Scene Recognition and Analysis; Based on the directional speech signal output in step 1.3, an unsupervised online acoustic scene clustering algorithm is used to perform feature analysis on the current environment, and obtain acoustic scene category identifiers and scene feature description vectors.

[0028] The purpose of acoustic scene recognition is to understand the current acoustic characteristics of the environment, providing a basis for subsequent adaptive noise reduction. Unlike traditional supervised classification methods, this invention employs an unsupervised clustering method, which can adaptively discover and learn new scene types, improving its adaptability to unknown environments. The specific implementation process is as follows: features reflecting environmental acoustic characteristics are extracted from directional speech signals, including short-time energy, zero-crossing rate, spectral centroid, spectral roll-off point, Mel-frequency cepstral coefficients, and other multi-dimensional features, forming a scene feature vector. An online clustering algorithm is used to analyze the feature vector. This algorithm maintains several cluster centers, each representing a typical acoustic scene. When a new feature vector arrives, its distance to each cluster center is calculated. If the minimum distance is less than a set threshold, the vector is assigned to the nearest cluster, and the cluster center is updated. If the minimum distance is greater than the threshold, a new cluster center is created, indicating the discovery of a new scene type. Through this mechanism, the system can continuously learn and adapt to changes in the deployment environment. The clustering process outputs a scene category identifier, using a numerical code to indicate which cluster it belongs to, and a scene feature description vector containing typical acoustic feature parameters of that scene.

[0029] Step 1.5, Adaptive noise reduction processing; Based on the scene category identifier and scene feature description vector output in step 1.4, a fast adaptive denoising neural network trained based on a meta-learning framework is used to perform noise suppression processing on the directional speech signal output in step 1.3, resulting in a clean speech signal for the target user with a high signal-to-noise ratio.

[0030] The noise reduction network employs an encoder-decoder architecture. The encoder consists of multiple layers of gated recurrent units (ROUs) responsible for extracting the temporal and noise features of the speech signal. The decoder, also composed of RROs, generates a spectral masking matrix based on the features extracted by the encoder. This masking matrix is ​​multiplied by the spectrum of the input signal to preserve speech components while suppressing noise. Finally, the time-domain waveform is reconstructed using an inverse short-time Fourier transform. The key innovation of this network lies in the introduction of a meta-learning mechanism: during training, the network not only learns how to reduce noise but also how to quickly adapt to new noise types and acoustic environments. Specifically, the training process simulates different acoustic scene tasks, each corresponding to a specific noise characteristic of a particular scene type. By iteratively training across multiple tasks, the network learns general noise reduction knowledge across tasks and task-specific rapid adaptation capabilities. In practical applications, based on the scene features identified in step 1.4, the network can quickly adjust its internal parameters using a small amount of current environmental data to specifically optimize the noise reduction strategy. The output of the noise reduction process is a clean speech signal for the target user, in which environmental noise components are effectively suppressed, speech components are enhanced and preserved, and the signal-to-noise ratio is improved compared to the original signal.

[0031] Step 1.6, Voice endpoint detection; Based on the clean speech signal of the target user output in step 1.5, the start and end times of valid speech are identified by an endpoint detection algorithm based on energy and spectral features, and a speech segment sequence is obtained.

[0032] Endpoint detection aims to segment segments containing speech information from a continuous audio stream, removing silence and pure noise to provide accurate input boundaries for subsequent recognition. The algorithm combines multiple criteria for decision-making: calculating the short-time energy and short-time zero-crossing rate of the signal; when the energy exceeds an energy threshold and the zero-crossing rate is within the typical range for speech, it is initially identified as a candidate speech start point; calculating the spectral characteristics of the signal, including frequency band energy distribution and harmonic structure features; speech signals typically have strong energy and exhibit harmonic structures in specific frequency bands, and the reliability of the energy criterion is verified through spectral characteristics; utilizing the continuity constraint between consecutive frames to avoid erroneous segmentation caused by instantaneous noise or speech pauses; and combining the results of various criteria to mark the start and end frames of the speech. The output is a sequence of speech segments, each containing a continuous user speech expression, along with start and duration information.

[0033] In some embodiments, since multiple people may speak simultaneously in a hospital environment, a deep learning-based multi-speaker detection and separation technique can be employed. The aim is to identify and separate the speech signals of different speakers. Specifically, based on the clean speech signal, a bidirectional long short-term memory network is used to estimate the number of speakers, obtaining the current number of speakers. Based on the number of speakers, a speech separation network trained using a permutation-invariant training method is used to separate the mixed speech, obtaining the independent speech signal of each speaker. Based on the spatial orientation information obtained in step 1.2, a spatial location matching strategy is used to select the separated speech signal that best matches the target user's location, obtaining the target user's clean speech signal. This alternative embodiment can effectively handle the complex situation of multiple people speaking simultaneously, improving the system's robustness in crowded environments.

[0034] In some embodiments, since the speaker broadcasts of the self-service machine may interfere with voice acquisition, acoustic echo cancellation technology can be employed. The aim is to eliminate echo interference formed when sound from the speaker travels through the air and is picked up by the microphone. Specifically, based on the reference audio signal played by the speaker and the audio signal picked up by the microphone, an adaptive filtering algorithm is used to estimate the acoustic echo path from the speaker to the microphone, obtaining an echo path model. Based on the echo path model and the reference audio signal, convolution operations are used to predict the echo components in the microphone-acquired signal, obtaining an estimated echo signal. Based on the microphone-acquired signal and the estimated echo signal, subtraction operations are used to eliminate the echo components, obtaining the echo-free voice signal. This alternative embodiment supports full-duplex interaction, allowing the system to receive user voice input while broadcasting information, improving the naturalness and efficiency of the interaction.

[0035] The speech recognition module performs speech-to-text recognition and confidence assessment based on speech segment sequences to obtain the recognized text sequence. Step 2.1, Acoustic feature extraction; Based on the speech segment sequence output in step 1.6, short-time Fourier transform and Mel filter bank processing are used to obtain the Mel spectrum feature sequence reflecting the time-frequency characteristics of speech.

[0036] Acoustic feature extraction converts time-domain speech signals into feature representations suitable for neural network processing. The specific process is as follows: The speech signal is pre-emphasized and filtered to compensate for energy attenuation of high-frequency components; the signal is framed, with each frame being 25 milliseconds long and shifted by 10 milliseconds, with overlap between adjacent frames to maintain temporal continuity; each frame is multiplied by a Hamming window function to reduce discontinuities at frame edges; the windowed signal is subjected to a Fast Fourier Transform to obtain a frequency domain representation; this frequency domain representation is passed through a set of Mel-scale triangular filter banks. The Mel scale simulates the nonlinear perception of frequency by the human ear, providing high resolution in the low-frequency region and low resolution in the high-frequency region; the filter bank typically contains 40 to 80 filters; the output energy of each filter is calculated and its logarithm is taken to obtain the log-Mel spectrum; the log-Mel spectra of all frames are arranged in chronological order to form a two-dimensional Mel spectrum feature matrix, where the row dimension corresponds to the frequency dimension and the column dimension corresponds to the time dimension.

[0037] Step 2.2, Local Feature Extraction; Based on the Mel spectrum feature sequence output in step 2.1, a convolutional neural network is used for local feature extraction and spectral pattern recognition to obtain a high-level acoustic representation.

[0038] Convolutional neural networks (CNNs) extract hierarchical feature representations from the Mel spectrum through multiple layers of convolution and pooling operations. Lower-level convolutional kernels capture local time-frequency patterns, such as formant structures and transition features; middle-level kernels combine lower-level features to form more complex patterns, such as phoneme-level acoustic features; higher-level kernels further abstract these features, forming acoustic representations corresponding to characters or words. The network structure contains multiple convolutional layers, each followed by a batch normalization layer and an activation function layer. Batch normalization accelerates training convergence and improves the model's generalization ability, while the activation function uses a modified linear unit to introduce non-linearity. Max-pooling layers are inserted between convolutional layers to downsample the feature maps, reducing feature dimensionality and enhancing feature invariance. The output of the convolutional network is a high-dimensional sequence of acoustic feature vectors, the length of which is related to the time dimension of the input, with one feature vector corresponding to each time step.

[0039] Step 2.3, Temporal Modeling and Context Aggregation; Based on the high-level acoustic representation output in step 2.2, a multi-layer bidirectional long short-term memory (BiLSTM) network is used for temporal modeling and contextual information aggregation to obtain a hidden state sequence that integrates the preceding and following context.

[0040] Bidirectional Long Short-Term Memory (LSTM) networks are effective models for processing sequential data, capable of capturing long-range temporal dependencies. The bidirectional structure comprises recurrent networks in both forward and backward directions: the forward network processes the sequence from left to right, capturing the influence of historical information; the backward network processes the sequence from right to left, capturing the influence of future information. The hidden states from both directions are concatenated at each time step to form a complete representation containing bidirectional context. Through the design of input, forget, and output gates, the LSM units can selectively remember important information and forget irrelevant information, effectively mitigating the gradient vanishing problem and supporting long sequence modeling. The network consists of multiple stacked bidirectional LSM layers, with the output of each layer serving as the input of the next, abstracting feature representations layer by layer. The final output sequence of hidden states contains rich acoustic and linguistic information, with the hidden state at each time step integrating contextual information within a certain range before and after that moment.

[0041] Step 2.4, sequence-to-sequence conversion; Based on the hidden state sequence output in step 2.3, an encoder-decoder architecture based on an attention mechanism is used to perform sequence-to-sequence conversion, and the probability distribution of the candidate text character sequence, the attention weight distribution during the decoding process, and the probability distribution entropy value are obtained.

[0042] The encoder-decoder architecture is a classic model for handling variable-length sequence mapping. The encoder part, as described above, consists of convolutional and recurrent networks, outputting a sequence of hidden state states as the encoded result. The decoder part generates the output character sequence step by step based on the encoded result. The decoding process is autoregressive: at each decoding step, the decoder predicts the probability distribution of the next character based on the generated character history, the current hidden state, and attention weights applied to the encoder output. The attention mechanism is a key component, allowing the decoder to dynamically focus on relevant parts of the encoder output when generating each character, rather than relying on a fixed encoded vector. Attention calculation includes: calculating the correlation score between the decoder's current state and the encoder's states at each time step, reflecting the importance of each encoding position to the current decoding; normalizing the score to obtain attention weights, with the weights summing to 1; weighting and summing the encoder states according to the attention weights to obtain a context vector, which aggregates the encoded information most relevant to the current decoding; combining the context vector with the decoder state, and generating the probability distribution of the next character through fully connected layers and activation functions. The decoding process is repeated until the end symbol is generated, outputting a complete candidate text sequence. To improve decoding quality, a beam search strategy is used instead of a greedy search. The beam search retains multiple candidate paths at each step and finally selects the path with the highest total probability as the output.

[0043] Step 2.5, re-scoring the language model; Based on the candidate text character sequence output in step 2.4, a language model that integrates medical business vocabulary and user expression habits is used for re-scoring to obtain the optimal recognized text result.

[0044] The language model evaluates the rationality and fluency of text sequences, supplementing the linguistic knowledge that the acoustic model fails to fully utilize. The language model used in this invention is obtained by further domain-adapted training on medical business-related texts based on a general Chinese corpus. The training corpus includes medical terminology, descriptions of common symptoms, department names, examination names, and doctor-patient dialogue records. The model adopts a Transformer architecture and captures long-distance dependencies in the text through a self-attention mechanism. In practical applications, combined with user profile information, the model can also adapt to the expression habits of specific users; for example, older users tend to use colloquial expressions, while younger users may use popular internet slang. The re-scoring process is as follows: the acoustic model provides candidate text sequences and their acoustic scores, the language model calculates the language model score for each candidate sequence, and the two are weighted and summed to obtain a total score. The candidate with the highest total score is selected as the optimal recognition result. The weight parameters are determined by optimizing the recognition accuracy on the validation dataset, balancing the contributions of the acoustic model and the language model.

[0045] Step 2.6, Confidence assessment; Based on the attention weight distribution and probability distribution entropy value output in the decoding process of step 2.4, a multi-dimensional confidence evaluation algorithm is used to calculate the confidence score of each recognized character, resulting in a recognized text sequence with character-level confidence annotations.

[0046] Confidence assessment quantifies the reliability of the recognition results, providing a basis for subsequent error detection and user confirmation. The evaluation algorithm integrates multiple indicators: character-level posterior probability, i.e., the probability value of the character output by the decoder; a higher probability indicates that the model is more confident in the prediction. Entropy value of the probability distribution, which measures the concentration of the probability distribution; low entropy indicates that the probability is concentrated on a few candidates, and the model's judgment is clear; high entropy indicates that the probability is scattered across multiple candidates, and the model's judgment is hesitant. Dispersion of attention weights; if the attention weights are concentrated on a few coding positions, it indicates that the character has clear acoustic support and high confidence; if the attention weights are scattered across many positions, it indicates that the acoustic information is unclear and low confidence. Consistency with the language model; if both the acoustic model and the language model give high scores, it indicates that the recognition result has both acoustic evidence and conforms to language rules, and high confidence. The algorithm combines these indicators through weighting or machine learning models to obtain a confidence score for each character, usually normalized to between 0 and 1; the closer the score is to 1, the higher the confidence. The output is a sequence of recognized text, with each character in the sequence accompanied by its confidence score, forming a recognition result with character-level confidence annotations.

[0047] In some embodiments, due to significant differences in the speech characteristics of different users, speaker adaptation technology can be employed to improve recognition accuracy. The aim is to fine-tune the recognition model parameters based on the current user's speech characteristics. Specifically, based on paired samples composed of the user's historical speech data and recognition results, a supervised fine-tuning method is used to update the parameters of some layers of the acoustic model to obtain a user-specific adapted model. The adapted model is then used to replace the general model for recognition, resulting in an optimized recognition result for that user. This replacement embodiment, through personalized adaptation, can better handle personalized features such as the user's accent, speech rate, and pronunciation habits, thereby improving recognition accuracy.

[0048] In some embodiments, due to potential identification errors, confusion networks and candidate fusion techniques can be employed to improve identification robustness. The aim is to retain multiple candidate identification results and select the better one in subsequent processing. Specifically, based on the decoder's beam search process, an N-optimal path retention strategy is used to obtain the top N candidate text sequences with the highest scores, where N is a preset number of candidates. Based on each candidate sequence and its probability score, a compact candidate space representation is constructed using a confusion network representation method, resulting in a candidate network structure. Based on the candidate network structure and feedback from the subsequent semantic understanding module, a candidate re-ranking strategy is used to select the candidate with the highest semantic consistency, obtaining the final identified text. This alternative embodiment, by retaining multiple candidates and delaying decision-making, can utilize more contextual and semantic information for selection, improving the overall system accuracy.

[0049] The user identification module identifies users based on speech segment sequences and recognized text sequences, obtaining user identification identifiers; and constructs personalized profiles based on user identification identifiers to obtain user configuration parameters. Step 3.1, Voiceprint Feature Extraction; Based on the speech segment sequence output by the audio acquisition module, a deep speaker embedding network is used to extract the user's voiceprint features, resulting in a fixed-dimensional voiceprint feature vector.

[0050] Voiceprints possess individual distinctiveness and relative stability. Different individuals have different vocal organ structures and pronunciation habits, resulting in unique acoustic characteristics in speech signals. Deep speaker embedding networks extract compact voiceprint representations from speech signals using deep learning. The network structure typically uses a time-delay neural network or residual network as the backbone to encode the input acoustic feature sequence. Hierarchical features are extracted through multiple nonlinear transformations. Variable-length feature sequences are aggregated using statistical pooling layers; common pooling methods include mean pooling, standard deviation pooling, or attention-weighted pooling, resulting in a fixed-dimensional speaker-level representation. The final speaker embedding vector is obtained through fully connected layers and normalization layers, typically with 256 or 512 dimensions. The network training employs metric learning, ensuring that embedding vectors from different speech segments of the same speaker are close in vector space, while embedding vectors from different speakers are far apart. During extraction, the clean speech signal undergoes the same acoustic feature extraction process to obtain Mel-spectral features, which are then input into the trained speaker embedding network for forward propagation to obtain the voiceprint feature vector corresponding to that speech segment.

[0051] Step 3.2, Voiceprint matching and identity recognition; Based on the voiceprint feature vector output in step 3.1, the cosine similarity metric is used to perform voiceprint matching in the local user database to obtain user identity identifiers or new user tags.

[0052] The user database stores the voiceprint templates and related information of registered users. The matching process is as follows: The cosine similarity between the current voiceprint feature vector and the voiceprint templates of each user in the database is calculated. Cosine similarity is defined as the inner product of two vectors divided by the product of their respective moduli, ranging from -1 to +1, with values ​​closer to 1 indicating greater similarity. All user templates in the database are traversed to find the user with the highest similarity. If the highest similarity exceeds a preset identity verification threshold, the current user is confirmed as a registered user, and their user identity identifier is returned. If the highest similarity is below the threshold, the current user is not in the database, is marked as a new user, and a new user identity identifier is assigned to them. To improve matching robustness and speed, the database can use an index structure such as an approximate nearest neighbor search algorithm for acceleration. The matching result includes the user identity identifier, which is associated with the user's personal information, historical interaction records, preference settings, and other data.

[0053] Step 3.3, User profile retrieval or construction; Based on the user identity identifier output in step 3.2, determine whether it is a new user: if it is a registered user, retrieve the user's profile data from the database to obtain user attributes and service preferences; if it is a new user, start the rapid profile building process.

[0054] For registered users, the database stores their complete user profile, including basic attributes such as age group, gender, and occupation, as well as behavioral data such as historical medical records, frequently used departments, language habits, and preferred interaction methods. Directly retrieving this data yields user attributes and service preference configurations for subsequent personalized processing.

[0055] For new users, due to the lack of historical data, it is necessary to quickly analyze the voice and text features of the current conversation to build an initial profile.

[0056] Step 3.4, inference of speech attributes; For new users, based on the acoustic features of the speech segment sequences output by the audio acquisition module, a speech attribute classification model is used to infer user attributes, resulting in estimated age group, gender, and emotional state.

[0057] Speech attribute classification models infer speaker attributes by analyzing the acoustic features of speech. Age inference is based on features such as fundamental frequency, formants, and voice quality. Different age groups have varying degrees of development and aging of their vocal organs, leading to differences in acoustic features. The model learns these patterns to classify speech and outputs age group labels such as child, youth, middle-aged, and elderly. Gender inference is based on fundamental frequency and spectral envelope features. Physiological differences in vocal tract length and vocal cord vibration frequency between men and women result in male voices typically having a lower fundamental frequency than female voices, and differences in spectral energy distribution. The model performs binary classification based on these differences. Emotional state inference is based on prosodic features such as pitch variation, speech rate, and energy dynamics. Different emotions, such as calmness, anxiety, and anger, affect the prosodic patterns of speech. The model identifies these patterns and assigns an emotion label. The classification model uses a convolutional neural network or long short-term memory network architecture and is trained on labeled data. Features are extracted from the current user's speech signal and input into the model to obtain the attribute inference results.

[0058] Step 3.5, Language Feature Analysis; For new users, based on the language features of the identified text sequence output in step 2.6, dialect recognition and language style analysis algorithms are used to obtain the user's dialect type and expression habits.

[0059] Dialect recognition analyzes dialect-specific vocabulary, grammatical structures, and expressions in text. People from different dialect regions are influenced by their dialects when using Mandarin, resulting in specific vocabulary and grammatical habits. By learning the text features of users from various dialect regions, the model can identify the user's possible dialect background, such as Northern Mandarin, Cantonese, Wu dialect, and Min dialect. Language style analysis includes dimensions such as formality, colloquialism, and education level, and is evaluated through features such as lexical richness, sentence complexity, and grammatical regularity. These analyses provide a reference for subsequent semantic understanding and response generation, enabling the system to better understand the user's expression and respond in an appropriate style.

[0060] Step 3.6, Basic configuration initialization; Based on the user attribute inference results output in step 3.4 and the dialect type output in step 3.5, the transfer learning method is used to select an initialization model from the model library of similar user groups to obtain the basic service configuration for new users.

[0061] The model library stores specialized models trained for different user groups, such as models for the elderly and models for specific dialect groups. Based on the attributes of new users, the most suitable group model is selected as the starting point. Transfer learning leverages the common knowledge of similar groups to provide new users with better initial performance than general models, mitigating the cold start problem. Basic service configurations include parameter settings for the recognition model, threshold adjustments for semantic understanding, and preferences for dialogue strategies.

[0062] Step 3.7, Dynamic Configuration Optimization; Based on the user's current session interaction data, including multi-turn voice and text, an online incremental learning algorithm is used to update the user profile and personalized model parameters, resulting in dynamically optimized user configuration parameters.

[0063] As users interact with the system, data from the current session accumulates, reflecting the user's true characteristics and needs. Incremental learning algorithms utilize this new data to update user profiles and models in real time. Specifically, the recognition model can be fine-tuned using few-shot learning techniques with the speech-text pairs of the current session, adjusting its adaptability to the user's voice features; preferences and habits in the user profile are updated based on user choices and feedback, such as marking a user's frequently asked questions about a particular department as an area of ​​interest; dialogue strategies are adjusted based on the user's response patterns, such as using more concise expressions when the user shows impatience with detailed explanations. Incremental learning is lightweight, requiring minimal data and computation, and can be performed in real time during the session. The output is a dynamically optimized user configuration, which becomes increasingly tailored to the user's characteristics as the interaction deepens.

[0064] In some embodiments, due to privacy protection requirements, the storage and use of voiceprint data and user profiles require strict security measures. Localized storage and encryption technologies can be employed to prevent the leakage of user privacy data. Specifically, based on user voiceprint features and profile data, advanced encryption standard algorithms are used for encrypted storage, resulting in ciphertext data. The key is controlled by the user or protected by a secure chip, preventing unauthorized decryption. The voiceprint matching process is performed in a local trusted execution environment, and the original voiceprint data is not uploaded to the cloud. A complete audit log is maintained for user data access and usage, and abnormal access behavior is checked regularly. This alternative embodiment ensures that user privacy is fully protected and complies with data protection regulations and ethical requirements.

[0065] In some embodiments, to further enhance the security of voiceprint recognition and prevent playback attacks, liveness detection technology can be employed. This aims to verify that the current voice is from a real, live user rather than a pre-recorded message. Specifically, based on the detailed features of the voice signal, high-frequency component analysis and phase consistency detection algorithms are used to obtain a liveness score. During playback, high-frequency details are lost and phase relationships change due to the secondary conversion by the speaker and microphone; these features can distinguish between liveness and pre-recorded messages. Combined with real-time verification during user interaction, a random challenge response mechanism is used, requiring the user to repeat randomly generated numbers or words to obtain a liveness confirmation result. This alternative embodiment improves the security level of identity authentication and prevents malicious attacks.

[0066] The semantic understanding module performs semantic understanding and intent recognition based on the identified text sequence and user identity identifier to obtain a structured user intent representation; Step 4.1, confidence score fusion text encoding; Based on the recognized text sequence and word-level confidence labels output in step 2.6, the text is encoded using a confidence-weighted text representation method to obtain a text vector representation that incorporates confidence information.

[0067] Traditional text encoding methods treat text as a deterministic sequence of characters, ignoring potential errors in speech recognition. This invention incorporates confidence information into the text representation, enabling subsequent semantic understanding to distinguish between high-confidence and low-confidence portions and to handle uncertain parts specially. Specifically, a pre-trained Chinese language model, such as BERT, is used to encode the text, obtaining context-dependent vector representations for each character. The confidence score of each character is used as a weight to perform weighted modulation on the vector representations; the representation vectors of low-confidence characters are attenuated, while those of high-confidence characters are retained. The weighted vector representation contains both the semantic information of the text and reflects the reliability of the recognition. This representation is used for subsequent intent classification and slot recognition tasks.

[0068] Step 4.2, Intent Classification and Slot Identification; Based on the text vector representation output in step 4.1, a deep neural network under a multi-task learning framework is used to perform intent classification and slot recognition, resulting in intent category labels and slot label sequences.

[0069] Intent classification determines the user's interaction purpose and falls under the category of text classification tasks. Deep neural networks map text vector representations to an intent category space using fully connected layers and activation functions. A normalized exponential function is used to obtain the probability distribution of each intent category, and the category with the highest probability is selected as the predicted intent. Intent categories are defined based on the services supported by the self-service machine, such as appointment booking, departmental consultation, examination appointment, report inquiry, fee inquiry, and complaints / suggestions.

[0070] Slot identification extracts key information entities from text and falls under the category of sequence labeling tasks. A slot represents a parameter required to fulfill an intent; for example, the slots needed for a registration intent include department, doctor, date, and time period. Deep neural networks label each character in the text, using labeling methods to distinguish the start, interior, and end positions of entities. Labeling networks typically employ conditional random fields or pointer networks, which can consider dependencies between labels and improve label consistency.

[0071] This multi-task learning framework trains both intent classification and slot recognition tasks simultaneously, sharing an underlying text encoder. Joint optimization enables the two tasks to mutually reinforce each other. Intent information helps slot recognition determine which entities are relevant, while slot information helps intent classification understand the user's specific needs. The deep neural network is trained on labeled data, derived from real user interaction records and manual annotations.

[0072] Step 4.3, Business rule verification; Based on the intent category and slot information output in step 4.2, a business rule verification algorithm is used to perform a logical consistency check, and potential conflicts and missing information are marked.

[0073] Business rules define the constraints between various intents and slots. For example, the intent to book an appointment must include a department slot, and optionally include doctor and date slots; some departments only operate at specific times; some examinations require specific preconditions, etc. The validation algorithm matches the identified intents and slots against the rule base to check whether the necessary conditions are met and whether there are any logical contradictions. For example, if a user says, "I want to book an emergency appointment for tomorrow," there is a logical conflict as emergency rooms typically do not offer appointment services, and the algorithm will mark this conflict. Similarly, if a user says, "Help me book an examination" but does not specify the examination, there is a missing key slot, and the algorithm will mark the missing item. The validation results are a list of potential conflicts and missing information, guiding subsequent dialogue management and information completion.

[0074] Step 4.4, Dialogue State Tracking; Based on the user identity and dialogue history output in step 3.2, a dialogue state tracking model is used to maintain the context state of multi-turn dialogues, resulting in a complete semantic representation that includes both referents and omissions.

[0075] In multi-turn dialogues, user expressions are often not isolated but based on the context of previous conversations. Users use pronouns such as "this" and "that" to refer to previously mentioned entities, omit known information, and correct and supplement previous content. The dialogue state tracking model maintains a dialogue state, recording currently confirmed slot values, candidate slot values, user intent history, and system behavior history. When new user input arrives, the dialogue state tracking model combines the current input and the dialogue state to infer, identify the referred entity, complete the omitted content, and update the slot value. For example, if a user says in the first turn, "I want to register for an appointment in cardiology," the system confirms the department as cardiology. In the second turn, if the user says, "Are there any appointments tomorrow morning?", the dialogue state tracking model understands that "tomorrow morning" is a time slot for registration, related to the intent to register for cardiology, and adds the time slot to the state. In the third turn, if the user says, "If not, then the day after tomorrow," the dialogue state tracking model understands this as a correction to the time slot and changes the time from tomorrow to the day after tomorrow. Dialogue state tracking models typically employ recurrent neural networks or memory network architectures, enabling them to maintain a long-term dialogue history. The output is a complete semantic representation, containing all relevant information accumulated from the multi-turn dialogue, forming a structured intent and slot representation.

[0076] Step 4.5, Knowledge Graph Validation; Based on the complete semantic representation output in step 4.4, common sense reasoning and medical knowledge graph verification are used to obtain semantic rationality scores and potential error prompts.

[0077] Even after grammatical and rule checks, semantic inconsistencies may still exist, requiring deeper knowledge-based reasoning for verification. Common-sense reasoning utilizes a common-sense knowledge base, containing everyday facts and rules such as "children usually see pediatricians" and "pregnant patients are women." The medical knowledge graph contains professional knowledge in the medical field, such as the relationships between diseases, symptoms, departments, and treatment methods. The verification process involves mapping entities and relationships in the complete semantic representation to the knowledge graph and checking for inconsistencies or contradictions. For example, a patient's attribute is male, but the registered department is obstetrics and gynecology, contradicting common sense; the symptom is palpitations, but the recommended department is ophthalmology, which is inconsistent with medical knowledge. The reasoning algorithm can be rule-based or neural network-based, judging reasonableness through logical reasoning or similarity calculation. The output is a semantic reasonableness score; a high score indicates semantic reasonableness, and a low score indicates potential problems. It also provides potential error hints, pointing out specific suspicious points, such as "gender does not match department" or "symptoms have low correlation with department." This information guides the system in confirming the interaction with the user.

[0078] Step 4.6, User Confirmation and Verification; Based on the semantic rationality score and potential error prompts output in step 4.5, an active learning strategy is adopted to determine whether confirmation from the user is required. If so, a targeted confirmation question is generated to obtain a structured user intent representation and key slot information that has been verified by the user.

[0079] The proactive learning strategy determines whether human feedback is needed based on uncertainty. When the semantic reasonableness score falls below a set threshold, or when multiple low-confidence characters exist in the identified text, the system judges that the current understanding may be incorrect and requires confirmation from the user. Confirmation questions are generated for specific points of suspicion, using natural language generation technology to express the questions in a user-friendly manner. For example, for the prompt "gender and department mismatch," the confirmation question is generated: "Are you trying to register for the obstetrics and gynecology department? Please note that this department primarily serves female patients"; for low-confidence time slots, the question is generated: "Do you mean tomorrow or the day after tomorrow?" The system broadcasts the questions to the user via speech synthesis and waits for a response. The user's response is processed by the speech recognition and semantic understanding modules to obtain confirmation or correction information, updating the semantic representation. The reliability of the semantic representation after user verification is improved, avoiding business processing failures due to misunderstandings.

[0080] In some embodiments, due to the specialized nature and colloquial expression in the medical field, there may be a large number of synonyms and non-standard terms. Medical terminology normalization techniques can be employed to map colloquial symptom descriptions to standard medical terminology. Specifically, based on symptom entities obtained through slot identification, a similarity matching algorithm is used to retrieve similar terms from a standard medical terminology database, resulting in a candidate standard terminology list. Based on contextual semantic information, a context-related reordering model is used to select the most suitable standard term, obtaining a normalized symptom representation. The normalized terms are used for subsequent knowledge graph reasoning and department recommendation, improving reasoning accuracy. This alternative embodiment bridges the gap between colloquial expression and standard medical knowledge, enhancing the professionalism of semantic understanding.

[0081] In some embodiments, to handle complex composite intents, a hierarchical intent parsing technique can be employed. This technique aims to decompose a complex expression containing multiple sub-intents into multiple independent intents. Specifically, based on text vector representation, a sequence segmentation algorithm is used to identify intent boundaries in the text, obtaining each intent fragment. Based on each intent fragment, independent intent recognition and slot extraction are performed to obtain multiple intent-slot pairs. Based on the relationships between intents, intent dependency parsing is used to determine the execution order and dependencies of each intent, resulting in a structured composite intent representation. For example, a user saying "I want to check the report results first, and then make an appointment for a follow-up checkup" contains two intents: querying the report and making an appointment, with a sequential dependency. This alternative embodiment can handle more complex user needs and improve the system's intelligence level.

[0082] The intelligent reasoning module performs intelligent reasoning and decision-making based on structured user intent representation and user identity identifiers, recommends departments and formulates treatment plans to obtain a complete treatment plan; Step 5.1, Medical Entity Linking; Based on the symptom and disease-related slots in the structured user intent representation output in step 4.6, a fuzzy entity linking algorithm is used to map colloquial expressions to standard entity nodes in the medical knowledge graph, resulting in a candidate medical entity set.

[0083] A medical knowledge graph is a structured knowledge base where nodes represent medical concepts such as diseases, symptoms, departments, examinations, and drugs, and edges represent relationships between entities, such as symptom-disease, disease-department, and disease-examination. Entity linking is the process of associating entity mentions in text with corresponding nodes in the knowledge graph. Because users use colloquial expressions that may differ from standard terminology in the knowledge graph, fuzzy matching is required. The algorithm first calculates string similarity between the entity mentioned by the user and the entity names in the knowledge graph, including metrics such as edit distance and pinyin similarity; it then expands the matching range using entity aliases and a thesaurus; combining contextual semantic information, it uses a semantic similarity model to calculate the distance between the user's mention and candidate entities in the semantic space; finally, it ranks the candidate entities based on both character and semantic similarity, selecting the top few as the candidate medical entity set. For example, if a user says "stomach ache," it links to the "abdominal pain" node in the knowledge graph; if a user says "heart discomfort," it links to candidate nodes such as "chest tightness" and "palpitations."

[0084] Step 5.2, Graph Neural Network Inference; Based on the candidate medical entity set output in step 5.1 and the attributes such as age and gender in the user configuration parameters output in step 3.7, a graph attention network is used to perform multi-hop relationship reasoning on the knowledge graph to obtain the probability distribution of related diseases and the list of recommended departments.

[0085] Graph Attention Networks (GANs) are deep learning models for processing graph-structured data, enabling information propagation and aggregation reasoning on knowledge graphs. The reasoning process involves using candidate symptom entities as starting nodes, propagating along symptom-disease relationships on the knowledge graph, and aggregating information from related disease nodes. Simultaneously, user attributes such as age and gender are considered, as these attributes are related to the epidemiological characteristics of certain diseases (e.g., some diseases are prevalent in specific age groups, and some only occur in specific genders). Attribute information is incorporated as a condition into the reasoning process. The graph attention mechanism automatically learns the importance weights of each relationship and node during propagation; important paths and nodes receive higher attention. The model learns these weights through training on labeled data. After multi-hop propagation, disease nodes aggregate information from symptoms, user attributes, and inter-disease relationships. A classification layer outputs a probability score for each disease; diseases with high scores represent the most likely diagnostic direction given the current symptoms and user attributes. Based on disease nodes, propagation further follows disease-department relationships to obtain a list of recommended departments. The department recommendation score comprehensively considers factors such as disease probability and departmental professional matching. The output is a sorted list of recommended departments and corresponding reasons for the recommendations.

[0086] Step 5.3, Incremental update of the knowledge graph; Based on the timestamps and source information of the medical knowledge graph, an incremental update mechanism is used to regularly synchronize the latest medical knowledge from authoritative medical databases and literature repositories to obtain a continuously evolving knowledge graph.

[0087] Medical knowledge is constantly evolving, with new diseases, treatments, and drugs emerging continuously, and existing knowledge may also be updated or revised. To maintain the timeliness and accuracy of the knowledge graph, the system employs an incremental update mechanism. The specific process is as follows: New knowledge is periodically acquired from authoritative sources such as treatment guidelines published by the National Health Commission, medical journal articles, and clinical databases; natural language processing and information extraction are performed on the new knowledge to identify entities and relationships; the new entities and relationships are compared with the existing knowledge graph to determine whether they are new additions or updates to existing knowledge; for new knowledge, it is directly added to the graph; for updated knowledge, version control and conflict resolution mechanisms are used to retain the source and timestamp information of the knowledge, and multiple versions are retained for comparison when necessary; a consistency check is performed on the graph to ensure that the updated graph is free of logical contradictions; and knowledge quality is periodically assessed, and low-quality or outdated knowledge is cleaned up or marked. Through incremental updates, the knowledge graph always reflects the latest medical advancements, providing a reliable basis for reasoning.

[0088] Step 5.4, Real-time resource synchronization; Based on the interface of the hospital information system, an event-driven real-time data synchronization mechanism is used to obtain dynamic resource information such as appointment slots, scheduling, and equipment status, and to obtain a real-time snapshot of the currently available resources.

[0089] Department recommendations need to be based on the hospital's actual resource situation, such as whether a department currently has available appointments, whether specialists are on duty, and whether examination equipment is operating normally. This information is dynamic and needs to be acquired in real time. The system connects to the hospital information system through standard interfaces such as HL7 or FHIR. An event-driven mechanism is adopted: when resource status changes, such as appointments being booked, doctors temporarily canceling their appointments, or equipment malfunctions, the hospital information system proactively pushes notification messages to the self-service machine system; the self-service machine system receives the messages and updates its local resource status cache; simultaneously, the system also periodically and proactively queries the status of key resources to ensure data synchronization. Resource information includes the availability of appointments for each department and doctor at each time slot, the number of people waiting, the estimated waiting time, the operating status of examination equipment, and appointment scheduling. A real-time snapshot shows the resource status at the current moment, used for subsequent route planning and appointment operations.

[0090] Step 5.5, Multi-objective path planning; Based on the recommended department list output in step 5.2, the real-time resource snapshot output in step 5.4, and the user's medical treatment constraints, a multi-objective constraint optimization algorithm is used to generate a personalized medical treatment path planning scheme, resulting in a complete medical treatment plan that includes department selection, time arrangement, examination items, and location navigation.

[0091] Medical route planning is a multi-objective optimization problem that requires comprehensive consideration of multiple factors: the credibility of department recommendations, the availability of appointment slots, user time preferences, convenience of medical visits, and the rationality of examination items. Constraints include explicitly stated user requirements such as specified time periods and doctors, as well as objective constraints such as available appointment slots and department opening hours. The optimization algorithm constructs an objective function, combining various optimization objectives with weights, such as minimizing consultation time, maximizing department matching, and maximizing user satisfaction. Weights are set based on user preferences and business strategies. The optimal solution is searched within the feasible solution space that satisfies the constraints. Heuristic algorithms such as genetic algorithms and simulated annealing can be used, or the solution can be modeled as an integer programming problem. The output medical route plan includes: recommended departments and doctors, suggested consultation times, required examination items and their order, locations and navigation routes for each step, and estimated total time. The plan is presented to the user in a structured and visual manner, allowing the user to accept it or request adjustments.

[0092] Step 5.6, Personalized service recommendations; Based on the user identity and historical medical records output in step 3.2, a collaborative filtering recommendation algorithm is used to make personalized service recommendations, resulting in customized service suggestions and quick operation access for the user.

[0093] Collaborative filtering is a classic method in recommendation systems, making recommendations based on the similarity of user behavior. The algorithm analyzes a user's historical medical records, including the department visited, frequency, and type of disease, to find other users with similar behavioral patterns and recommend services frequently used by these similar users but not currently used by the current user. For example, a user with a chronic disease who regularly visits a certain department can be provided with quick registration for that department, follow-up appointment reminders, and chronic disease management suggestions. Simultaneously, content-based recommendation methods analyze the user's health records and current needs to recommend relevant health education content and preventative healthcare services. The recommendation results are integrated into the user interface of the self-service machine as personalized menus or shortcut buttons, improving user efficiency and experience.

[0094] In some embodiments, due to the complexity and uncertainty of disease diagnosis, reasoning based solely on knowledge graphs may be insufficient. Deep learning-based diagnostic assistance models can be combined to improve the accuracy of disease prediction and department recommendation. Specifically, based on user symptom descriptions and structured health record data, a multimodal fusion deep neural network is used for end-to-end disease prediction to obtain the probability distribution of diseases. This network is trained on large-scale electronic medical record data, learning the complex mapping relationships between symptoms, signs, test results, and diseases. The prediction results of the deep learning model are fused with the results of knowledge graph reasoning, and an ensemble learning method is used to obtain a comprehensive disease probability assessment. Departments are recommended based on the comprehensive assessment results. This alternative embodiment combines the advantages of both knowledge-driven and data-driven methods, improving the accuracy and robustness of reasoning.

[0095] In some embodiments, to provide more comprehensive medical advice, a multidisciplinary consultation recommendation mechanism can be introduced. The aim is to recommend multidisciplinary joint treatment plans for complex conditions. Specifically, based on the disease probability distribution, a complexity assessment algorithm is used to determine whether the condition requires multidisciplinary collaboration, obtaining a complexity score. When the complexity exceeds a threshold, a multidisciplinary association analysis algorithm is used to identify the necessary departmental combinations in a knowledge graph, resulting in a recommended list of consultation departments. Based on the hospital's multidisciplinary consultation resources and processes, a consultation appointment plan is generated and recommended to the user. This alternative embodiment provides more professional medical advice for difficult and complex cases, improving the quality of medical services.

[0096] The dialogue management module, based on structured user intent representation and complete medical treatment plan, performs multi-round dialogue management and information completion to obtain complete business execution instructions; Step 6.1, Slot integrity check; The business process specifications are obtained from the hospital information system. Based on the structured user intent representation output in step 4.6 and the complete medical treatment plan in step 5.5, a slot integrity and consistency check algorithm is used to obtain a list of missing slots and conflicting slot markers.

[0097] Each business intent corresponds to a set of required slots and optional slots. Required slots contain essential information for completing the business operation; without them, execution is impossible. Optional slots provide auxiliary information to improve service quality; even without them, execution may not be as effective. The integrity check iterates through the list of required slots for the current intent, checking if each slot is filled. Unfilled slots are added to the missing slot list. The consistency check verifies whether there are logical contradictions between the slot values, such as an appointment date earlier than the current date or a time period that does not match the department's opening hours. Slots with contradictions are marked with a conflict flag. The output is a list of missing slots and conflicting slot flags, guiding subsequent inquiry and confirmation strategies.

[0098] Step 6.2, Dialogue Strategy Selection; Based on the missing and conflicting slot information output in step 6.1, a hierarchical dialogue strategy network is used to select the optimal interaction action to obtain targeted inquiry or suggestion dialogue.

[0099] The dialogue strategy determines the actions the system should take in the current dialogue state, such as asking for missing information, confirming conflicting information, providing suggested options, or executing business operations. The hierarchical policy network includes high-level and low-level policies: high-level policies are responsible for the macro-planning of the dialogue flow, such as determining which slot information should be collected, whether explanation or assistance should be provided to the user, etc.; low-level policies are responsible for selecting specific interactive actions, such as using different methods for the same missing slot, such as open-ended questions, providing options for the user to choose from, or providing default values ​​based on user profiles. The policy network is trained through reinforcement learning: the process of system-user interaction is modeled as a Markov decision process, where the state is the current dialogue state including collected slots, user attributes, and dialogue rounds; actions are optional interactive actions; and a reward function evaluates the success and efficiency of the dialogue, awarding higher rewards for successful completion of business operations with fewer rounds. The policy network learns to select actions that maximize long-term rewards in each state. Training data comes from real user interactions and simulated dialogues. In practical applications, the policy network outputs action decisions based on the current state, such as "ask for appointment dates."

[0100] Step 6.3, Natural Language Generation; Based on the action decision output in step 6.2, a combination of template filling and neural language generation is used to generate natural and fluent interactive dialogue, resulting in user-friendly inquiry or confirmation statements.

[0101] Script generation transforms abstract actions into concrete natural language expressions. For common standardized inquiries, a template-filling method is used. Predefined script templates, such as "Which day would you like to book?", are filled in with variables like slot names, resulting in fast and standardized generation. For complex explanatory or suggestive expressions, a neural language generation model is employed. This model is based on a Transformer decoder architecture, taking the current dialogue state and action type as input and outputting natural language sentences. The model is pre-trained on a large-scale dialogue corpus, learning diverse expressions and styles to generate fluent, polite, and context-appropriate scripts. The generation process allows control over attributes such as formality and detail, and personalization based on user profiles. For example, simpler and more direct expressions can be used for older users, while a more colloquial style can be used for younger users. The generated scripts undergo grammar checking and sensitive word filtering to ensure quality and security.

[0102] Step 6.4, Multimodal Interaction Output; Based on the generated interactive script output in step 6.3, neural speech synthesis technology is used to convert the text into natural speech signals, which are then played to the user through the speaker of the self-service machine. At the same time, the text and related graphical interfaces are displayed on the screen, resulting in multimodal interactive output.

[0103] Speech synthesis converts text into speech, enabling users to acquire information through hearing. An end-to-end neural speech synthesis model is employed, directly generating speech waveforms from text. This model comprises an acoustic model and a vocoder: the acoustic model converts text into acoustic features such as Mel-frequency spectra, using a sequence-to-sequence architecture, with the encoder encoding the text and the decoder generating a sequence of spectral frames using an attention mechanism; the vocoder converts the Mel-frequency spectra into temporal waveforms, employing generative adversarial networks or streaming models to generate high-fidelity audio. The model is trained on a large amount of speech data, synthesizing accurate, rhythmic, and expressive speech. The synthesis process allows control over parameters such as speaker timbre, speech rate, and pitch. The system can select a friendly timbre and a moderate speech rate to enhance the user experience. The synthesized speech signal is output through the self-service machine's speaker. Simultaneously, the text is displayed on the screen with large fonts and high contrast for easy reading by users with poor eyesight; related graphical interfaces such as a date picker, department list, and map navigation are also updated synchronously, and users can perform assisted operations via the touchscreen. Multimodal output meets the preferences and needs of different users, improving the accessibility and convenience of interaction.

[0104] Step 6.5, User Response Processing; Based on the user's voice response, the audio acquisition module, speech recognition module, user identification module, and semantic understanding module are repeatedly executed to obtain the recognized text sequence and semantic understanding results of the user's response, and to extract supplementary or corrected slot information.

[0105] After hearing the system's inquiry, the user provides information via voice response. The system collects the user's voice and, through signal enhancement, speech recognition, and semantic understanding, obtains recognized text and semantic representation. Slot values ​​are extracted and filled into the corresponding slots. If the user's response includes a correction to previous information, such as "not tomorrow, but the day after tomorrow," the system recognizes the correction intent and updates the corresponding slot. If the user's response contains new intents or multiple pieces of information, the system switches intents or processes them in parallel. The extracted slot information is then merged into the current structured intent representation.

[0106] Step 6.6, Update the dialogue status; Based on the newly extracted slot information and dialogue history output in step 6.5, a dialogue state backtracking and conflict resolution mechanism is adopted to obtain a consistent dialogue state and an updated intent representation.

[0107] New information may conflict with previous information. Conflict detection compares the new slot value with historical slot values; if an inconsistency is found, the conflict resolution process is initiated. Conflict resolution strategies include: asking the user to clarify their intent, such as, "You previously mentioned cardiology, but now you've said gastroenterology. Which department would you like to register with?"; replacing older information with the latest information based on time priority; and retaining information with high confidence levels. State backtracking checks key decision points in the dialogue history to ensure consistency between the current state and historical logic. After resolving the conflict, the dialogue state and intent representation are updated to obtain a consistent version.

[0108] Step 6.7, iterative information collection; Based on the updated intent representation output in step 6.6, repeat steps 6.1 to 6.6 until all necessary slots are filled completely and without conflict, thus obtaining a complete business execution instruction.

[0109] The process iteratively collects and confirms information, checking the integrity and consistency of slots in each round. If incomplete, it continues to query until the business execution conditions are met. A maximum number of dialogue rounds is set to prevent infinite loops; if the process is still incomplete after exceeding the maximum number of rounds, the user is prompted to switch to human assistance or simplify their request. Upon completion, a business instruction containing all necessary information is output, which can be directly passed to the business execution module.

[0110] Step 6.8, Dialogue Quality Assessment; Based on the complete and user-confirmed business execution instructions output in step 6.7 and the interaction logs of the entire dialogue process, the success rate and user experience of this dialogue are evaluated using a dialogue quality assessment model to obtain a quality score and improvement suggestions.

[0111] The dialogue quality assessment measures dialogue effectiveness from multiple dimensions: task completion (whether the required information was successfully collected and business operations executed); dialogue efficiency (the number of rounds and time required to complete the task); user satisfaction (assessed based on user tone, emotion, and clear feedback); and interaction naturalness (whether the dialogue is fluent and the wording appropriate). The assessment model calculates a quality score by integrating these indicators. It also analyzes problems during the dialogue process, such as repeated questions due to identification errors and inefficiencies caused by inappropriate strategy selection, generating improvement suggestions. The scores and suggestions are used for subsequent model optimization: the dialogue strategy is trained using the scores as reward signals through reinforcement learning; an error detection model is trained using labeled problem samples through supervised learning; and continuous iterative improvement is achieved using user feedback data.

[0112] In some embodiments, to handle complex user expressions and multiple intent inputs, dynamic intent switching and parallel slot filling techniques can be employed. The aim is to flexibly address diverse user needs in single-turn or multi-turn dialogues. Specifically, based on semantic understanding results, an intent switching detection algorithm identifies whether the user has changed their current intent, obtaining an intent switching flag. If a switch is detected, an intent stack management mechanism is used to push the current intent onto the stack for storage, and then the user switches to the new intent for processing, resulting in dialogue management that supports intent nesting and backtracking. If the user's expression contains multiple parallel intents, an intent separation and independent processing strategy is adopted, collecting slot information for each intent separately, and finally merging them for execution. This alternative embodiment improves the flexibility of dialogue management and can handle complex situations in real user interactions.

[0113] In some embodiments, to improve the naturalness of dialogue and user experience, affective computing and affective response mechanisms can be introduced. The aim is to perceive user emotions and provide appropriate emotional responses. Specifically, based on the prosodic features of the user's speech, such as pitch, speech rate, and energy changes, an affective recognition model is used to determine the user's emotional state, obtaining emotion categories such as calm, anxiety, and anger, and their intensity. Based on the recognized emotional vocabulary and tone of the text, text affective analysis is used to supplement the emotion judgment. Based on the user's emotional state, an affective-adaptive dialogue strategy is adopted: when the user is anxious, reassurance and patient explanation are provided; when the user is angry, apologies are expressed and the problem is quickly resolved, resulting in emotionally resonant interactive dialogue. During speech synthesis, the prosodic parameters of the synthesized speech are adjusted according to the context, so that the system's speech expression also carries appropriate emotional coloring. This alternative embodiment enhances the emotional resonance of human-computer interaction, improving user satisfaction and trust.

[0114] The business execution module performs privacy protection processing based on complete business execution instructions, and allocates and executes tasks to obtain business execution results and multimodal user feedback. Step 7.1, Privacy Information Identification and Labeling; Based on the user-sensitive information involved in the complete business execution instruction output in step 6.7, privacy identification is performed using named entity recognition and information classification algorithms to obtain data items labeled with sensitivity levels.

[0115] Business instructions contain sensitive information such as users' names, ID numbers, and descriptions of their medical conditions, which need to be identified and categorized. The algorithm iterates through each data item in the instruction, classifying it according to predefined sensitive information categories such as identity information, health information, and financial information, and labeling it with a sensitivity level such as high sensitivity, medium sensitivity, and low sensitivity. Information that directly identifies an individual, such as ID numbers and medical record numbers, is considered highly sensitive; health information such as symptom descriptions and examination results is considered medium sensitive; and general business information such as the registration department and appointment time is considered low sensitive. The labeling results guide subsequent privacy protection measures.

[0116] Step 7.2, Data privacy protection processing; Based on the sensitive data items output in step 7.1, differential privacy or data anonymization techniques are used for privacy protection processing to obtain the protected data.

[0117] Highly sensitive information is handled using encryption or local processing strategies. For example, ID numbers are transmitted using homomorphic encryption, or identity is verified only locally without uploading the original data. Moderately sensitive information is anonymized; for instance, specific details in medical condition descriptions are generalized or obfuscated, retaining key features for business processing while removing personally identifiable details. Low-sensitivity information can be used directly or subjected to simple obfuscation. Differential privacy technology adds carefully designed noise to the data, ensuring that the presence or absence of a single record does not affect statistical results, protecting individual privacy while allowing for data analysis. The processed data can be used for business execution while minimizing the risk of privacy breaches.

[0118] Step 7.3, Task allocation decision; Based on the type and computational requirements of the complete business execution instructions, a dynamic task allocation algorithm is used to determine whether to execute on the local end, edge server, or cloud, resulting in a task allocation scheme.

[0119] Different business operations have varying computational complexity and data sensitivity, necessitating intelligent allocation of computing nodes. The allocation algorithm considers the following factors: computational complexity (simple queries and rule matching are executed locally, while complex inference and big data analysis are performed in the cloud); real-time requirements (urgent tasks are prioritized for local or edge execution to reduce latency, while non-urgent tasks can be scheduled to the cloud); data sensitivity (highly sensitive data is processed locally first, while low-sensitivity data can be uploaded to the cloud); network conditions (cloud resources can be utilized when the network is good, while local resources are used only when the network is poor or down); and resource load (the current load of each node is considered to avoid overloaded nodes). The algorithm integrates these factors, employing heuristic rules or machine learning models to make decisions and output a plan on which node each subtask should be executed on. Lightweight models and core business logic are deployed locally; medium-sized models and cached business data are deployed on edge servers; and large-scale models and a complete database are deployed in the cloud.

[0120] Step 7.4, Distributed task execution; Based on the task allocation scheme output in step 7.3 and the protected data output in step 7.2, a secure communication protocol is used to transmit data and execute tasks between computing nodes to obtain the processing results of each node.

[0121] Data transmission employs transport layer security protocols or virtual private networks (VPNs) for encryption to prevent eavesdropping or tampering during transmission. Each node executes its assigned tasks in parallel or sequentially; for example, local nodes perform user authentication and data preprocessing, edge servers handle business logic processing and database queries, and the cloud performs complex reasoning and decision support. During execution, nodes communicate via message queues or remote procedure calls to exchange intermediate results. After completing their tasks, each node returns processing results, including business status, returned data, and execution logs.

[0122] Step 7.5, Results Integration and Decision Making; Based on the processing results of each node output in step 7.4, the results are integrated, consistency is verified, and a final decision is made locally to obtain the final result of business execution.

[0123] The local endpoint acts as the master node, aggregating the results returned by each node. The integration process includes data merging, format conversion, and logical judgment. Consistency checks examine for conflicts or anomalies between results; for example, if one node returns success but another fails, the overall business status needs to be assessed. The final decision, based on business rules and results, determines whether the operation was successful, whether remedial measures are needed, and what information to send to the user. The output is the final result of the business execution; for example, a successful appointment includes the appointment number, appointment time, and location, while a failed appointment includes the reason for failure and suggestions.

[0124] Step 7.6, Multimodal content generation; Based on the business execution results output in step 7.5, natural language generation technology is used to construct result description text, and information visualization technology is used to design a graphical interface to obtain multimodal feedback content.

[0125] The system transforms structured business results into user-friendly expressions. Natural language generation selects appropriate templates or generation strategies based on the result type. For example, a successful appointment generates "You have successfully booked an appointment with a doctor in a specific department on a specific date. Please pick up your number and see a doctor at a specific location before a specific time." A failed appointment generates "We're sorry, this department is currently full. We suggest you choose another time or department." The text is clear, polite, and includes necessary information and guidance. Information visualization presents key information graphically, such as flowcharts, hospital map navigation, and QR code vouchers, making it intuitive and easy to understand. The interface design follows accessibility principles, with appropriate font sizes, high color contrast, and a simple layout, making it easy for the elderly and visually impaired to use.

[0126] Step 7.7, User Feedback Output; Based on the feedback output in step 7.6, speech synthesis technology is used to generate voice broadcasts, visual content is displayed on the screen, and the printer is controlled to output paper vouchers or QR codes according to the business type, thus obtaining complete multimodal user feedback.

[0127] Speech synthesis converts the textual description of the results into speech, which is then played through a speaker, allowing users to obtain the results auditorily. The display screen simultaneously shows text and graphics, providing visual information. For services requiring proof, such as appointment booking and payment, the system controls a printer to output paper vouchers containing QR codes, medical information, and precautions; users can then use these paper vouchers or scan the QR code with their mobile phones at other points within the hospital. Multimodal output ensures users receive comprehensive information, while paper vouchers provide convenience for users who do not use smartphones.

[0128] Step 7.8, Operation Audit Record; Based on the data from the entire interaction process, a distributed logging system is used to record the operation trajectory and system status. Data anonymization and access control are used to protect log security, resulting in a complete audit record.

[0129] Logs record all stages of user interactions, including timestamps, user identifiers, operation types, data inputs and outputs, system responses, and abnormal events. Logs are used for troubleshooting, performance analysis, security auditing, and service quality monitoring. Sensitive information in the logs is anonymized, removing or obfuscating identity and privacy data while retaining only essential information for analysis. Logs are stored on a secure server using access control policies, allowing only authorized personnel access when necessary, and all access operations are recorded. Logs are regularly backed up and archived to meet data retention and compliance requirements.

[0130] A voice recognition-based intelligent interactive self-service machine is provided, which stores computer-readable instructions and can run a voice recognition-based intelligent interactive self-service system as described above when the computer-readable instructions are read by a computer.

[0131] In one embodiment of the present invention, a specific example is provided: The intelligent interactive self-service machine and system based on voice recognition of this invention have been successfully deployed and applied in a tertiary hospital. The hospital has an average daily outpatient volume of over 10,000 visits, an environmental noise level of 65-75 decibels, and a diverse patient population, with elderly patients over 60 years old accounting for approximately 40%. The hospital deployed 4 intelligent voice interactive self-service machines in the outpatient hall and 6 machines in the triage areas on each floor, covering the main medical procedures.

[0132] Example 1: Symptom Description and Department Recommendation for an Elderly Patient. A 75-year-old female patient, speaking with a regional accent, described her symptoms: "My stomach has been uncomfortable for the past few days; it's a bit bloated and painful, and I have no appetite." The system, through speech enhancement, semantic understanding, and knowledge graph reasoning, accurately identified the symptoms and recommended a gastroenterology department, generating the following feedback: "Based on your description of abdominal discomfort, bloating, and loss of appetite, we recommend you schedule an appointment with a gastroenterologist. We found that there are specialist appointments available at 3 PM today and general appointments available at 9 AM tomorrow. Which time would you like to choose?" Example 2: Multi-round dialogue for information completion. A middle-aged male patient said, "I want to have an MRI." The system gradually completed the information, such as the examination site (head), the prescribing doctor (neurologist Dr. Zhang), and the appointment time, through 5 rounds of dialogue, and finally successfully completed the appointment in less than 2 minutes.

[0133] Table 1 shows examples of interaction data from different user groups: Table 1: Examples of interaction data for different user groups;

[0134] Table 2 shows examples of data processed in typical business scenarios: Table 2: Examples of processing data in typical business scenarios;

[0135] After three months of actual deployment, the system has served over 50,000 patients. Key achievements include: speech recognition accuracy reaching practical levels even in noisy hospital environments; high accuracy in understanding common patient needs; completion of most simple tasks within three rounds, and an average of five rounds for complex tasks; and high user satisfaction, particularly among elderly and visually impaired users.

[0136] This invention promotes the intelligentization and accessibility of medical services, lowers the barriers to medical care, reduces the risk of disease transmission, improves medical efficiency, and optimizes the allocation of hospital human resources. The technical solution can be extended to other self-service fields such as government services, banking, transportation, and shopping malls, and has broad application prospects and commercial value.

[0137] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A voice recognition-based intelligent interactive self-service system, characterized in that, include: The audio acquisition module is used to acquire multi-channel audio signals, extract the speech signals of the target user, and perform signal processing to obtain a speech segment sequence. The speech recognition module performs speech-to-text recognition and confidence assessment based on speech segment sequences to obtain the recognized text sequence. The user identification module identifies users and obtains user identity identifiers based on speech segment sequences and recognized text sequences. Personalized profiles are constructed based on user identity identifiers to obtain user configuration parameters; The semantic understanding module performs semantic understanding and intent recognition based on the identified text sequence and user identity identifier to obtain a structured user intent representation; The intelligent reasoning module performs intelligent reasoning and decision-making based on structured user intent representation and user identity identifiers, recommends departments and formulates treatment plans to obtain a complete treatment plan; The dialogue management module, based on structured user intent representation and complete medical treatment plan, performs multi-round dialogue management and information completion to obtain complete business execution instructions; The business execution module performs privacy protection processing based on complete business execution instructions, and allocates and executes tasks to obtain business execution results and multimodal user feedback.

2. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The acquisition of multi-channel audio signals includes: Based on a circular array of six omnidirectional electret condenser microphones evenly distributed on the front frame of the self-service machine, multi-channel digital acquisition is performed to obtain the raw audio data stream. Based on the original audio data stream, the generalized cross-correlation time delay estimation algorithm is used to calculate the arrival time difference of the target speech signal received by each microphone channel, and to obtain the estimated spatial azimuth and distance of the target user relative to the center of the microphone array. Based on spatial azimuth and distance estimates, an adaptive minimum variance distortion-free response beamforming algorithm is used to perform spatial filtering on multi-channel audio signals to obtain directional speech signals.

3. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The steps to obtain a sequence of speech segments include: Based on directional speech signals, an unsupervised online acoustic scene clustering algorithm is used to perform feature analysis on the current environment, and obtain acoustic scene category identifiers and scene feature description vectors. Based on acoustic scene category identifiers and scene feature description vectors, noise suppression processing is performed on directional speech signals to obtain clean speech signals for the target user. Based on the clean speech signal of the target user, an endpoint detection algorithm based on energy and spectral features is used to identify the start and end times of valid speech, and obtain a speech segment sequence.

4. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The speech recognition process includes: Based on the speech segment sequence, short-time Fourier transform and Mel filter bank processing are used to obtain the Mel spectral feature sequence that reflects the time-frequency characteristics of speech. Based on the Mel spectral feature sequence, a convolutional neural network is used for local feature extraction and spectral pattern recognition to obtain a high-level acoustic representation. Based on high-level acoustic representation, a multi-layer bidirectional long short-term memory network is used for temporal modeling and contextual information aggregation to obtain a hidden state sequence that integrates the preceding and following context. Based on the hidden state sequence, an encoder-decoder architecture based on an attention mechanism is used to perform sequence-to-sequence transformation to obtain the probability distribution of candidate text character sequences; Based on the candidate text character sequence, a language model that integrates medical business vocabulary and user expression habits is used for re-scoring to obtain the optimal recognized text result.

5. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The confidence assessment includes calculating the confidence score of each recognized character based on the attention weight distribution and probability distribution entropy value during the decoding process, using a multi-dimensional confidence assessment algorithm to obtain the recognized text sequence, which is labeled with word-level confidence.

6. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The steps for obtaining the user's identity identifier include: Based on speech segment sequences, a deep speaker embedding network is used to extract the user's voiceprint features, resulting in a fixed-dimensional voiceprint feature vector. Based on the voiceprint feature vector, the cosine similarity measurement method is used to perform voiceprint matching in the local user database to obtain user identity identifiers or new user tags. Based on user identification, the system detects user registration status. If the user is already registered, it retrieves user profile data from the database to obtain user attributes and service preferences. If the user is a new user, it initiates a rapid profile building process.

7. The intelligent interactive self-service system based on speech recognition according to claim 6, characterized in that, The rapid profile building process includes: For new users, based on the acoustic features of speech segment sequences, a speech attribute classification model is used to infer user attributes, and the estimated age group, gender, and emotional state are obtained. For new users, based on the language features of the identified text sequence, dialect recognition and language style analysis algorithms are used to obtain the user's dialect type and expression habits. Based on user attribute inference results and dialect type, a transfer learning method is used to select an initialization model from a model library of similar user groups to obtain the basic service configuration for new users.

8. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The steps for obtaining a structured representation of user intent include: Based on the recognition of text sequences and word-level confidence annotations, a confidence-weighted text representation method is used to encode the text, resulting in a text vector representation that incorporates confidence information. Based on text vector representation, a deep neural network under a multi-task learning framework is used to perform intent classification and slot recognition, resulting in intent category labels and slot label sequences. Based on user identity and dialogue history, a dialogue state tracking model is used to maintain the context state of multi-turn dialogues and obtain a complete semantic representation.

9. The intelligent interactive self-service system based on speech recognition according to claim 1, characterized in that, The steps to obtain a complete medical plan include: Based on the symptom and disease-related slots in the structured user intent representation, a fuzzy entity linking algorithm is used to map colloquial expressions to standard entity nodes in the medical knowledge graph, resulting in a candidate medical entity set. Based on the candidate medical entity set and user configuration parameters, a graph attention network is used to perform multi-hop relationship reasoning on the knowledge graph to obtain the probability distribution of related diseases and the recommended department list. Based on the recommended department list, real-time resource snapshots, and user's medical treatment constraints, a multi-objective constraint optimization algorithm is used to generate a personalized medical treatment path planning scheme, resulting in a complete medical treatment plan that includes department selection, time arrangement, examination items, and location navigation.

10. A smart interactive self-service machine based on voice recognition, characterized in that, It is used to store computer-readable instructions, which, when read by a computer, enable the operation of a voice recognition-based intelligent interactive self-service system as described in any one of claims 1-9.

Images