Voice interaction recognition method and system for medical accompanying robot based on internet of things

Abstract

The application discloses a medical accompanying robot voice interaction recognition method and system based on the Internet of Things, which comprises the following steps: step one, collecting continuous audio stream and multi-source sensing data of a ward to construct a target patient sound interval model; step two, performing frame processing on the continuous audio stream, extracting sound source space features and sound physiological modulation features, and screening voice segments to generate real sound event segments; step three, collecting vital sign data, performing time alignment, and constructing a synchronous time window; step four, generating a disease course semantic set; step five, inputting voice features into an improved Paraformer model to perform multi-source semantic driven decoding processing to generate a candidate semantic set; step six, calculating a nursing trigger credibility; and step seven, performing hierarchical determination to generate an interaction instruction set. The application realizes accurate recognition and intelligent response of voice interaction in a complex medical scene.

Description

Technical Field

[0001] This invention relates to the fields of IoT medical and voice recognition technology, and in particular to a voice interaction recognition method and system for IoT-based medical companion robots. Background Technology

[0002] With the development of intelligent medical devices and the gradual application of medical companion robots, voice-based human-computer interaction has become an important means to improve nursing efficiency and patient experience. In ward settings, patients expressing their needs through voice and having the system automatically recognize and respond has become a research hotspot. Existing technologies mainly use deep learning-based speech recognition models to process the collected audio signals and combine them with simple rule or keyword matching methods to generate interactive commands. However, the following problems are common in actual medical environments:

[0003] The ward environment contains multiple sources of interference, such as conversations among medical staff, noise from equipment operation, and voices from other patients. Traditional speech recognition methods lack spatial constraint mechanisms for specific patients, making it difficult to accurately distinguish the target patient's voice from background interference, resulting in a high false recognition rate. Existing methods typically rely solely on the speech signal itself for processing, without incorporating information on changes in the patient's vital signs, and cannot assist in judging the validity of speech events from a physiological perspective, leading to insufficient recognition capabilities for abnormal calls or emergency situations. Furthermore, most existing speech recognition models adopt a single-path decoding structure, lacking enhancement mechanisms for medical semantics, making it difficult to fully utilize semantic information such as nursing level, disease information, and treatment tasks for recognition optimization. The recognition accuracy drops significantly in scenarios with weak speech, non-standard expressions, and semantic ambiguity. At the same time, existing systems often use simple threshold judgment or rule matching methods in the recognition result output stage, lacking a fusion decision-making mechanism for speech features, semantic information, and physiological state, making it difficult to achieve refined hierarchical responses and the generation of multiple types of instructions.

[0004] Therefore, how to provide a voice interaction recognition method and system for IoT-based medical companion robots is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a voice interaction recognition method and system for medical companion robots based on the Internet of Things. This invention achieves voice spatial constraints and interference suppression by constructing a target patient's vocal range model; extracts real vocal events by combining sound source features and physiological modulation features, and introduces a vital sign synchronization time window to enhance voice validity judgment; constructs a semantic set by integrating nursing level, disease type, and diagnosis and treatment information, and implements multi-source semantic-driven decoding in an improved Paraformer model; finally, it calculates the credibility of nursing triggers and performs graded judgment by fusing multi-source information, thereby improving the accuracy of voice recognition and the reliability of interactive response.

[0006] The voice interaction recognition method for medical companion robots based on the Internet of Things according to embodiments of the present invention includes the following steps:

[0007] Step 1: Collect continuous audio streams from the ward, as well as data on bed pressure, human presence, location tags, and door magnetic sensors. Construct a target patient's vocal range model based on bed coordinates and the patient's activity range.

[0008] Step 2: Perform frame segmentation processing on the continuous audio stream, extract the spatial features of the sound source and the physiological modulation features of vocalization, and combine the target patient's vocalization interval model to filter speech segments and generate real vocalization event segments.

[0009] Step 3: Collect vital sign monitoring data and treatment status data, perform time alignment and identify abrupt change points, construct a vital sign synchronization time window, and map the actual vocalization event fragments to the vital sign synchronization time window;

[0010] Step 4: Generate a disease course semantic set based on the nursing level, disease label, treatment task, and the synchronous time window status of the vital signs;

[0011] Step 5: Extract speech feature sequences from the real speech event segments and input them into the improved Paraformer model. Combine the pathological semantic set with the speech feature sequence to perform multi-source semantic-driven feature reconstruction and decoding processing to generate a candidate semantic set.

[0012] Step Six: Calculate the nursing trigger credibility based on the actual vocal event fragments, candidate semantic set, and vital sign synchronization time window;

[0013] Step 7: Perform hierarchical judgment processing based on the nursing trigger credibility, candidate semantic set, and the status category corresponding to the vital signs synchronization time window to generate an interactive instruction set.

[0014] Optionally, step one includes:

[0015] Read the pressure distribution data output by the bed pressure sensor, construct the bed surface pressure matrix according to the preset sampling period, and calculate the pressure concentration area, pressure center location and pressure distribution boundary based on the bed surface pressure matrix;

[0016] Read the occupancy status data output by the human body presence sensor to determine whether the patient is in bed or out of bed, and read the positioning tag data to determine the patient's real-time spatial coordinates;

[0017] Read the door magnetic data to determine whether the door is open or closed, and define the doorway activity area based on the door's position coordinates and a preset spatial range;

[0018] Under a unified ward coordinate system, coordinate alignment is performed on the bed center coordinates, pressure center location, pressure distribution boundary, real-time spatial coordinates of the patient, occupancy status, and doorway activity area.

[0019] The patient's main distribution area is calculated based on the bed center coordinates and the pressure center location; the posture offset area is calculated based on the pressure distribution boundary and the patient's real-time spatial coordinates; and the external interference area is determined based on the doorway activity area.

[0020] Under a unified ward coordinate system, the patient's main distribution area, posture deviation area, and external interference area are combined to construct a target patient vocal range model that includes the main vocal range, the deviation vocal range, and the inhibition area.

[0021] Optionally, step two includes:

[0022] The continuous audio stream is divided into frames according to a preset frame length and frame shift to obtain an audio frame sequence, and windowing processing is performed on each audio frame.

[0023] Based on the multi-channel audio data collected by the microphone array, cross-correlation calculation is performed on each audio frame to determine the time delay corresponding to the peak value of the cross-correlation function, and the sound source direction angle and sound source distance information are calculated based on the time delay.

[0024] Short-time energy sequences are calculated for each audio frame. The energy difference between adjacent audio frames is calculated and normalized to obtain the energy change rate. The fundamental frequency trajectory is extracted and the difference change between consecutive fundamental frequency frames is calculated. At the same time, bandpass filtering is performed on the low-frequency envelope to extract the periodic fluctuation amplitude.

[0025] Under the unified ward coordinate system, the sound source direction angle and sound source distance information are converted into spatial coordinate positions, and it is determined whether the spatial coordinate position falls into the main sound area or the offset sound area corresponding to the target patient's sound interval model.

[0026] The audio frames that fall within the target patient's vocal range model are counted sequentially in time. Audio frame sequences with a number of consecutive frames greater than the frame count threshold are spliced ​​together to generate real vocal event segments.

[0027] Optionally, step three includes:

[0028] The vital signs monitoring data and treatment status data are timestamped according to a unified time base to form a multidimensional time series arranged in chronological order.

[0029] The difference between each parameter in the multidimensional time series at adjacent sampling times is taken as the change amount. The absolute value of the change amount is processed and compared with the change threshold of the corresponding parameter one by one. When the change amount of at least two consecutive sampling times is greater than the corresponding change threshold, the starting sampling time of the continuous sampling time interval is determined as the change point.

[0030] Using the timestamp corresponding to the mutation point as the center, the time range is extended forward and backward respectively to construct a time window for synchronizing vital signs;

[0031] The time range of the actual speech event segment is overlapped with the time window of the vital signs synchronization. The ratio of the overlap time length to the duration of the speech segment is calculated, and the actual speech event segment with a ratio greater than the ratio threshold is identified as the synchronized speech segment.

[0032] Optionally, step four includes:

[0033] Read nursing level information, divide the nursing level into multiple level categories according to the preset level range, and number the semantic words corresponding to each level category to form a basic semantic index set;

[0034] Read the disease label information, find the corresponding set of symptom descriptive words based on the disease label, and perform vector encoding on each semantic word in the set of symptom descriptive words to form a set of disease semantic vectors;

[0035] Read the diagnosis and treatment task information, divide the diagnosis and treatment task into multiple task states according to the execution stage, and number the set of operation words corresponding to each task state to form a task semantic index set.

[0036] The system reads the state of the synchronous time window of vital signs, and determines the rising state of parameters with positive differences, the falling state of parameters with negative differences, and the stable state of parameters with changes less than the stability threshold, based on the sign of the difference between parameters corresponding to the mutation point at adjacent sampling times. The system also encodes the physiological response semantic words corresponding to different states to form a physiological semantic set.

[0037] The basic semantic index set, disease semantic vector set, task semantic index set, and physiological semantic set are concatenated, and duplicate values ​​are removed according to the semantic number to generate the disease course semantic set.

[0038] Optionally, step five includes:

[0039] Pre-emphasis, framing, windowing, and Mel spectrum transform are performed on real speech event segments to construct speech feature sequences. The speech feature sequences, disease course semantic sets, and vital signs synchronous time window states are then input into the improved Paraformer model.

[0040] The improved Paraformer model includes a sequentially connected multi-source semantic driving unit, encoder, predictor, dual-path decoder, and semantic difference unit. The encoder includes a multi-layer stacked feedforward module, self-attention module, and convolution module. The dual-path decoder includes a basic decoding path and a semantic enhancement decoding path.

[0041] The multi-source semantic driving unit performs numbering, sorting, and vector encoding on the semantic items in the disease course semantic set, constructs a semantic encoding sequence, and maps the state of the vital signs synchronization time window into a state encoding vector.

[0042] The multi-source semantic driving unit copies the state encoding vector along the time axis to the same length as the speech feature sequence according to the frame order of the speech feature sequence, expands the semantic encoding sequence according to the frame order and maps it to the same length and feature dimension as the speech feature sequence, and performs frame-by-frame splicing, linear transformation and element-wise weighting on the speech feature sequence, state encoding vector and semantic encoding sequence to construct the semantic driving feature sequence.

[0043] The encoder performs multi-layer temporal encoding on the semantically driven feature sequence. In each layer, the semantically driven feature sequence is input into the self-attention module and the convolution module respectively. The output of the self-attention module and the output of the convolution module are summed and then input into the feedforward module to form the encoded temporal representation.

[0044] The predictor performs length prediction and position alignment on the encoded temporal representation to construct the basic decoded input sequence;

[0045] The basic decoding path performs character-level decoding processing on the basic decoding input sequence to construct the basic text sequence;

[0046] The semantic enhancement decoding path concatenates and performs linear transformation on the semantic encoding sequence corresponding to the pathological semantic set and the basic decoding input sequence according to the position alignment result of the predictor output to construct the semantic enhancement decoding sequence, and performs character-level decoding processing on the semantic enhancement decoding sequence to construct the semantic enhancement text sequence;

[0047] The semantic difference unit performs position alignment processing on the basic text sequence and the semantically enhanced text sequence, identifies the difference character segments and difference word segments, extracts the newly added semantic segments, the replaced semantic segments and the common semantic segments, and performs filtering and recombination processing according to the semantic numbers in the disease course semantic set to construct a candidate semantic set.

[0048] Optionally, step six includes:

[0049] Speech intensity features and speech duration features are extracted from real speech event segments. Speech intensity features are calculated by summing the squares of the amplitude values ​​of each audio frame and then performing normalization processing. Speech duration is calculated by multiplying the number of consecutive frames by the frame length.

[0050] For each semantic item in the candidate semantic set, find the corresponding weight coefficient according to the semantic number, and perform weighted summation on each semantic item according to the semantic category to calculate the semantic score.

[0051] Normalize the changes of each parameter in the synchronous time window of vital signs, find the corresponding state weight according to the direction and magnitude of change, perform weighted summation on the state weights of each parameter, and calculate the physiological state score.

[0052] The speech intensity features, speech duration features, semantic score and physiological state score are weighted and summed, and the weights of each item are allocated according to the proportional coefficient to calculate the nursing trigger confidence value.

[0053] The nursing trigger confidence value is compared with the confidence threshold, and the result of the nursing trigger confidence value being greater than the confidence threshold is determined as a valid trigger event.

[0054] Optionally, step seven includes:

[0055] Read the nursing trigger credibility, candidate semantic set and the status category corresponding to the vital signs synchronization time window, and find the corresponding category label for each semantic item in the candidate semantic set according to the semantic number to construct a semantic category sequence;

[0056] The credibility of the nursing trigger is compared with the credibility grading threshold range to determine the range in which the nursing trigger credibility is located, and the status category is determined according to the status flag corresponding to the change direction of each parameter in the vital signs synchronization time window.

[0057] Based on the interval of nursing trigger credibility, semantic category sequence and state category, a combined judgment vector is constructed, and a hierarchical mapping process is performed on the combined judgment vector to determine the interaction level;

[0058] Each interaction level is pre-configured with a corresponding set of instruction numbers, and each set of instruction numbers includes at least one of voice broadcast instructions, device control instructions, nursing reminder instructions, and alarm reporting instructions.

[0059] Find the corresponding set of instruction numbers based on the interaction level, and execute and sort the instructions according to their numbers to generate a set of interactive instructions.

[0060] According to an embodiment of the present invention, a voice interaction and recognition system for a medical companion robot based on the Internet of Things includes:

[0061] The vocal range construction module is used to collect continuous audio streams from the ward, as well as bed pressure, human presence, location tags, and door magnetic data. Based on the bed coordinates and the patient's activity range, it constructs a vocal range model for the target patient.

[0062] The vocal event extraction module is used to perform frame-by-frame processing on the continuous audio stream, extract the spatial features of the sound source and the physiological modulation features of vocalization, and combine the target patient's vocal interval model to filter speech segments and generate real vocal event segments.

[0063] The vital signs synchronization module is used to collect vital signs monitoring data and treatment status data, perform time alignment and identify abrupt changes, construct a vital signs synchronization time window, and map real vocal event fragments to the vital signs synchronization time window.

[0064] The semantic generation module is used to generate a set of disease course semantics based on nursing level, disease label, diagnosis and treatment tasks, and the synchronous time window status of vital signs.

[0065] The speech recognition module is used to extract speech feature sequences from real speech event segments and input them into the improved Paraformer model to perform multi-source semantic-driven feature reconstruction and decoding processing to generate a candidate semantic set.

[0066] The credibility calculation module is used to calculate the credibility of nursing triggers based on real vocal event fragments, candidate semantic sets, and vital sign synchronization time windows.

[0067] The judgment and execution module is used to perform hierarchical judgment processing based on the nursing trigger credibility, candidate semantic set, and the status category corresponding to the vital signs synchronization time window, and generate a set of interactive instructions.

[0068] The beneficial effects of this invention are:

[0069] This invention constructs a target patient's vocalization interval model, integrating multi-source sensing information such as bed pressure, human presence, location tags, and door magnets to achieve spatial constraint and interference suppression of the target patient's vocalization area in the ward environment. Combining sound source spatial features and vocal physiological modulation features, it performs multi-dimensional feature filtering and continuity determination on continuous audio streams to generate realistic vocal event fragments, improving the accuracy of speech event extraction. In the temporal dimension, it introduces a vital sign synchronization time window, aligning and associating speech events with physiological states through differential calculation of vital sign changes and identification of abrupt change points, enhancing the ability to determine the validity of speech events. In the semantic modeling stage, it integrates nursing level, disease label, and treatment task information to construct a multi-source speech model. A semantically driven disease process semantic set is used to achieve semantic prior guidance in the speech recognition process. At the model level, an improved Paraformer structure is designed. Through the synergistic effect of multi-source semantic driving units, dual-path decoding mechanism and semantic difference unit, speech features are reconstructed and enhanced, significantly improving the recognition ability of complex semantics in medical scenarios. In the decision-making stage, a nursing trigger credibility calculation mechanism that integrates speech features, semantic information and physiological state is constructed. Combined with a hierarchical judgment strategy, an interactive instruction set is generated. This enables spatial perception, temporal synchronization, multi-source semantic fusion and intelligent decision processing of the medical companion robot's voice interaction, effectively improving the accuracy of speech recognition and the reliability of interactive response in complex ward environments. Attached Figure Description

[0070] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0071] Figure 1 This is a schematic diagram of the voice interaction recognition method for medical companion robots based on the Internet of Things proposed in this invention;

[0072] Figure 2 This is a schematic diagram of the structure of the Internet of Things-based voice interaction and recognition system for medical companion robots proposed in this invention;

[0073] Figure 3 This is a schematic diagram of the improved Paraformer model structure in the IoT-based voice interaction recognition method for medical companion robots proposed in this invention. Detailed Implementation

[0074] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0075] refer to Figures 1-3 A voice interaction recognition method for IoT-based medical companion robots includes the following steps:

[0076] Step 1: Collect continuous audio streams from the ward, as well as data on bed pressure, human presence, location tags, and door magnetic sensors. Construct a target patient's vocal range model based on bed coordinates and the patient's activity range.

[0077] Step 2: Perform frame segmentation on the continuous audio stream, extract the spatial features of the sound source and the physiological modulation features of vocalization, and combine the target patient's vocalization interval model to select speech segments and generate real vocalization event segments.

[0078] Step 3: Collect vital sign monitoring data and treatment status data, perform time alignment and identify abrupt changes, construct a vital sign synchronization time window, and map real vocalization event fragments to the vital sign synchronization time window;

[0079] Step 4: Generate a semantic set of disease progression based on nursing level, disease label, treatment task, and the synchronous time window status of vital signs;

[0080] Step 5: Extract speech feature sequences from real speech event segments and input them into the improved Paraformer model. Combine the pathological semantic set to perform multi-source semantic-driven feature reconstruction and decoding processing to generate a candidate semantic set.

[0081] Step Six: Calculate the credibility of nursing triggers based on real vocal event fragments, candidate semantic sets, and vital sign synchronization time windows;

[0082] Step 7: Perform hierarchical judgment processing based on the nursing trigger credibility, candidate semantic set, and the status category corresponding to the vital signs synchronization time window to generate an interactive instruction set.

[0083] In this embodiment, step one includes:

[0084] Read the pressure distribution data output by the bed pressure sensor, construct the bed surface pressure matrix according to the preset sampling period, and calculate the pressure concentration area, pressure center location and pressure distribution boundary based on the bed surface pressure matrix;

[0085] Read the occupancy status data output by the human body presence sensor to determine whether the patient is in bed or out of bed, and read the positioning tag data to determine the patient's real-time spatial coordinates;

[0086] Read the door magnetic data to determine whether the door is open or closed, and define the doorway activity area based on the door's position coordinates and a preset spatial range;

[0087] Under a unified ward coordinate system, coordinate alignment is performed on the bed center coordinates, pressure center location, pressure distribution boundary, real-time spatial coordinates of the patient, occupancy status, and doorway activity area.

[0088] The patient's main distribution area is calculated based on the bed center coordinates and the pressure center location; the posture offset area is calculated based on the pressure distribution boundary and the patient's real-time spatial coordinates; and the external interference area is determined based on the doorway activity area.

[0089] Under a unified ward coordinate system, the patient's main distribution area, posture deviation area, and external interference area are combined to construct a target patient vocal range model that includes the main vocal range, the deviation vocal range, and the inhibition area.

[0090] In this implementation, the sampling period of the bed pressure matrix is ​​set to 0.05s, the pressure sensor resolution is a 32×16 array, the pressure concentration area is determined by clustering cells with continuous pressure values ​​greater than 20kPa, the pressure center position is calculated by weighted average coordinates of pressure values ​​of each cell, and the pressure distribution boundary is determined by boundary cells with pressure values ​​less than 5kPa; the human presence sensor uses millimeter-wave radar, with a detection distance range set to 0.3m to 2.5m; the spatial positioning error of the positioning tag is controlled within ±0.1m; the doorway activity area is defined as a fan-shaped area with a radius of 1.2m, based on the coordinates of the door center; the unified ward coordinate system has the bed center as the origin, the length direction as the X-axis, and the width direction as the Y-axis; the patient's main distribution area is determined by extending 0.3m outward from the pressure center position, the posture deviation area is determined by extending 0.5m outward from the pressure distribution boundary, and the external interference area is limited to a range greater than 1.5m from the bed center.

[0091] In this embodiment, step two includes:

[0092] The continuous audio stream is divided into frames according to a preset frame length and frame shift to obtain an audio frame sequence, and windowing processing is performed on each audio frame.

[0093] Based on the multi-channel audio data collected by the microphone array, cross-correlation calculations are performed on each audio frame to determine the time delay corresponding to the peak value of the cross-correlation function, and the sound source direction angle and sound source distance information are calculated based on the time delay.

[0094] Short-time energy sequences are calculated for each audio frame. The energy difference between adjacent audio frames is calculated and normalized to obtain the energy change rate. The fundamental frequency trajectory is extracted and the difference change between consecutive fundamental frequency frames is calculated. At the same time, bandpass filtering is performed on the low-frequency envelope to extract the periodic fluctuation amplitude.

[0095] Under the unified ward coordinate system, the sound source direction angle and sound source distance information are converted into spatial coordinate positions, and it is determined whether the spatial coordinate positions fall into the main sound area or the offset sound area corresponding to the target patient's sound interval model.

[0096] The audio frames that fall within the target patient's vocal range model are counted sequentially in time. Audio frame sequences with a number of consecutive frames greater than the frame count threshold are spliced ​​together to generate real vocal event segments.

[0097] In this embodiment, audio framing uses a frame length of 25ms and a frame shift of 10ms, with a Hamming window applied to each frame; the cross-correlation calculation between microphone array channels uses the maximum correlation peak corresponding to the time delay, with a time delay resolution of 0.01ms; the sound source direction angle is calculated based on an array spacing of 0.05m, and the sound source distance is limited to the range of 0.3m to 3.0m; short-time energy is obtained by summing the squares of the amplitude of each frame, and the energy change rate is calculated using the ratio of the energy difference between adjacent frames to the energy of the previous frame; the fundamental frequency extraction range is set to 80Hz to 300Hz, and the threshold for the fundamental frequency difference between consecutive frames is set to 20Hz; the low-frequency envelope is extracted for periodic fluctuations through a bandpass filter from 20Hz to 80Hz; the frame count threshold in consecutive frame counting is set to the range of 5 to 15 frames, and a valid speech segment is determined when the number of consecutive frames is greater than 8 frames.

[0098] In this embodiment, step three includes:

[0099] The vital signs monitoring data and treatment status data are timestamped according to a unified time base to form a multidimensional time series arranged in chronological order.

[0100] The difference between each parameter in the multidimensional time series at adjacent sampling times is taken as the change amount. The absolute value of the change amount is processed and compared with the change threshold of the corresponding parameter one by one. When the change amount of at least two consecutive sampling times is greater than the corresponding change threshold, the starting sampling time of the continuous sampling time interval is determined as the change point.

[0101] Using the timestamp corresponding to the mutation point as the center, the time range is extended forward and backward respectively to construct a time window for synchronizing vital signs;

[0102] The time range of the actual speech event segment is overlapped with the time window of the vital signs synchronization. The ratio of the overlap time length to the duration of the speech segment is calculated, and the actual speech event segment with a ratio greater than the ratio threshold is identified as the synchronized speech segment.

[0103] In this implementation, the sampling period for vital sign monitoring data is set to 1 second, and the timestamp alignment error is controlled within ±0.02 seconds. Differential calculation is performed based on the difference between data from adjacent sampling times, and the absolute value of the change is used for comparison. The threshold for heart rate change is set to 15 bpm, the threshold for blood oxygen change is set to 3%, the threshold for respiratory rate change is set to 5 breaths / min, the threshold for body temperature change is set to 0.5℃, and changes in infusion status are determined based on a drip rate change greater than 20%. The time window is formed by extending forward by 5 seconds and backward by 8 seconds from the point of change. The duration of the speech segment is calculated by accumulating the frame length of 25 ms, and the overlap time length is the intersection length of the speech segment and the time window. The ratio is calculated by the ratio of the overlap time length to the duration of the speech segment. The ratio threshold is set to 0.6, and when the ratio is greater than 0.6, it is determined to be a synchronous speech segment.

[0104] In this embodiment, step four includes:

[0105] Read nursing level information, divide the nursing level into multiple level categories according to the preset level range, and number the semantic words corresponding to each level category to form a basic semantic index set;

[0106] Read the disease label information, find the corresponding set of symptom descriptive words based on the disease label, and perform vector encoding on each semantic word in the set of symptom descriptive words to form a set of disease semantic vectors.

[0107] Read the diagnosis and treatment task information, divide the diagnosis and treatment task into multiple task states according to the execution stage, and number the set of operation words corresponding to each task state to form a task semantic index set.

[0108] The system reads the state of the synchronous time window of vital signs, and determines the rising state of parameters with positive differences, the falling state of parameters with negative differences, and the stable state of parameters with changes less than the stability threshold, based on the sign of the difference between parameters corresponding to the mutation point at adjacent sampling times. The system also encodes the physiological response semantic words corresponding to different states to form a physiological semantic set.

[0109] The basic semantic index set, disease semantic vector set, task semantic index set, and physiological semantic set are concatenated, and duplicate values ​​are removed according to the semantic number to generate the disease course semantic set.

[0110] In this implementation, nursing levels are divided into 1 to 4, and the semantic word set corresponding to each level is determined by a lookup table, with the number of semantic words ranging from 10 to 50. The disease label adopts the standard disease classification code, and the symptom word set corresponding to the code is vectorized, with the vector dimension set to 64 to 128 dimensions. The diagnosis and treatment task is divided into preparation, execution and completion stages according to the execution process, with the number of operation words corresponding to each stage ranging from 5 to 20. The direction of change of vital signs is determined by the sign of the difference between adjacent sample values, and the stable state is determined based on the change being less than a set threshold, with the stable threshold being 20% ​​of the normal fluctuation range of the corresponding parameter. During the semantic set splicing process, the semantic set is sorted according to the semantic number, and a linear scanning method is used to remove duplicate semantic items to ensure that each semantic item in the output set is unique and retains the original category information.

[0111] In this embodiment, step five includes:

[0112] Pre-emphasis, framing, windowing, and Mel spectrum transform are performed on real speech event segments to construct speech feature sequences. The speech feature sequences, disease course semantic sets, and vital signs synchronous time window states are then input into the improved Paraformer model.

[0113] The improved Paraformer model includes a sequentially connected multi-source semantic driving unit, encoder, predictor, dual-path decoder, and semantic difference unit. The encoder includes a multi-layer stacked feedforward module, self-attention module, and convolution module. The dual-path decoder includes a basic decoding path and a semantic enhancement decoding path.

[0114] The multi-source semantic driving unit performs numbering, sorting, and vector encoding on the semantic items in the disease course semantic set, constructs a semantic encoding sequence, and maps the state of the vital signs synchronization time window into a state encoding vector.

[0115] The multi-source semantic driving unit copies the state encoding vector along the time axis to the same length as the speech feature sequence according to the frame order of the speech feature sequence, expands the semantic encoding sequence according to the frame order and maps it to the same length and feature dimension as the speech feature sequence, and performs frame-by-frame splicing, linear transformation and element-wise weighting on the speech feature sequence, state encoding vector and semantic encoding sequence to construct the semantic driving feature sequence.

[0116] The encoder performs multi-layer temporal encoding on the semantically driven feature sequence. In each layer, the semantically driven feature sequence is input into the self-attention module and the convolution module respectively. The output of the self-attention module and the output of the convolution module are summed and then input into the feedforward module to form the encoded temporal representation.

[0117] The predictor performs length prediction and position alignment on the encoded temporal representation to construct the basic decoded input sequence;

[0118] The basic decoding path performs character-level decoding processing on the basic decoding input sequence to construct the basic text sequence;

[0119] The semantic enhancement decoding path concatenates and performs linear transformation on the semantic encoding sequence corresponding to the pathological semantic set and the basic decoding input sequence according to the position alignment result of the predictor output to construct the semantic enhancement decoding sequence, and performs character-level decoding processing on the semantic enhancement decoding sequence to construct the semantic enhancement text sequence;

[0120] The semantic difference unit performs position alignment processing on the basic text sequence and the semantically enhanced text sequence, identifies the difference character segments and difference word segments, extracts the newly added semantic segments, the replaced semantic segments and the common semantic segments, and performs filtering and recombination processing according to the semantic numbers in the disease course semantic set to construct a candidate semantic set.

[0121] In this implementation, the semantic coding sequence in the multi-source semantic driving unit is generated using a lookup table method, with the coding dimension set to 64 to 128 dimensions. The state coding vector is transformed to the same dimension as the speech features through linear mapping. The speech features are modulated using a dimension-wise product method for element-wise weighted processing. The position alignment in the predictor is achieved by repeating or pruning the encoded temporal representation based on the length prediction result. In the dual-path decoding, the two paths share encoder parameters, and semantic coding differences are introduced only at the decoding input. In the semantic difference processing, the character position correspondence is established through sequence alignment, and the difference fragments are extracted using a sliding window method with a window length set to 2 to 5 characters to ensure the stability and continuity of the candidate semantic set.

[0122] The improved Paraformer model maintains the same overall structure as the Paraformer model, both consisting of three parts: encoder, predictor, and decoder. The encoder is used to perform temporal modeling on the input speech feature sequence, the predictor is used to generate length information and perform position alignment, and the decoder is used to convert the encoded features into a text sequence. The encoder adopts a structure that combines self-attention modules and convolutional modules to simultaneously obtain global dependencies and local temporal features, ensuring the ability to model speech sequences.

[0123] Building upon this foundation, the improved Paraformer model adds a multi-source semantic driving unit to the input side. This unit encodes the disease course semantic set and the synchronous time window state of vital signs, and performs alignment expansion, concatenation, and element-wise weighting processing on the speech feature sequence to construct a semantic driving feature sequence. Simultaneously, a dual-path decoding structure is set up on the decoding side, adding a semantic enhancement decoding path in addition to the basic decoding path. A semantic difference unit is introduced at the output end to align and extract differences between the two path outputs, forming a candidate semantic set.

[0124] Through the above improvements, speech features are integrated with disease course semantics and vital sign status information before entering the encoder, enabling speech representation to be dynamically adjusted according to changes in the medical scenario; the dual-path decoding structure provides two semantic expression results, and extracts key semantic information through differential processing, thereby improving the recognition stability of weak speech and non-standard expressions, and improving the recognition accuracy of semantics related to the medical scenario.

[0125] In this embodiment, step six includes:

[0126] Speech intensity features and speech duration features are extracted from real speech event segments. Speech intensity features are calculated by summing the squares of the amplitude values ​​of each audio frame and then performing normalization processing. Speech duration is calculated by multiplying the number of consecutive frames by the frame length.

[0127] For each semantic item in the candidate semantic set, find the corresponding weight coefficient according to the semantic number, and perform weighted summation on each semantic item according to the semantic category to calculate the semantic score.

[0128] Normalize the changes of each parameter in the synchronous time window of vital signs, find the corresponding state weight according to the direction and magnitude of change, perform weighted summation on the state weights of each parameter, and calculate the physiological state score.

[0129] A weighted summation process is performed on speech intensity features, speech duration features, semantic score values, and physiological state score values. The weights of each item are allocated according to a proportional coefficient, and the nursing trigger confidence value is calculated.

[0130] The nursing trigger confidence value is compared with the confidence threshold, and the result of the nursing trigger confidence value being greater than the confidence threshold is determined as a valid trigger event.

[0131] In this implementation, a dual-index mapping relationship is established between semantic IDs and weight coefficients. The first index is the semantic category, and the second index is the nursing level. The weight coefficient ranges from 0.2 to 1.0. The weight value of each semantic item in the candidate semantic set is calculated by multiplying the weight coefficient by the frequency of occurrence. The weight values ​​of each semantic item are accumulated to form a semantic score value, which ranges from 0 to 50. The changes in each parameter within the synchronous time window of vital signs are linearly normalized according to the normal fluctuation range of the corresponding parameter. The normalization result ranges from 0 to 1. The direction of change is divided into positive and negative based on the sign of the difference between adjacent samples. The magnitude of change is divided into three levels according to 0.3 and 0.7. Different levels correspond to state weights of 0.3, 0.6, and 0.9. The state weight of each parameter is multiplied by the parameter category coefficient (ranging from 0.5 to 1.5) and accumulated to form a physiological state score value. The credibility of nursing triggers is calculated by linear combination of speech intensity features, speech duration features, semantic score and physiological state score, with proportional coefficients of 0.15, 0.10, 0.35 and 0.40, respectively, and the credibility threshold is set to 0.6.

[0132] In this embodiment, step seven includes:

[0133] Read the nursing trigger credibility, candidate semantic set and the status category corresponding to the vital signs synchronization time window, and find the corresponding category label for each semantic item in the candidate semantic set according to the semantic number to construct a semantic category sequence;

[0134] The credibility of the nursing trigger is compared with the credibility grading threshold range to determine the range in which the nursing trigger credibility is located, and the status category is determined according to the status flag corresponding to the change direction of each parameter in the vital signs synchronization time window.

[0135] Based on the interval of nursing trigger credibility, semantic category sequence and state category, a combined judgment vector is constructed, and a hierarchical mapping process is performed on the combined judgment vector to determine the interaction level;

[0136] Each interaction level is pre-configured with a corresponding set of instruction numbers, and each set of instruction numbers includes at least one of voice broadcast instructions, device control instructions, nursing reminder instructions, and alarm reporting instructions.

[0137] Find the corresponding set of instruction numbers based on the interaction level, and execute and sort the instructions according to their numbers to generate a set of interactive instructions.

[0138] In this implementation, semantic category labels are obtained by indexing semantic numbers to a predefined semantic category table. Different semantic categories correspond to different priority levels, which are used to participate in subsequent hierarchical judgments. The combined judgment vector is formed by concatenating nursing trigger confidence interval codes, semantic category codes, and status category codes in a fixed order, and then performing normalization processing on each code item before participating in weighted calculations. The hierarchical mapping adopts an interval matching method to map the combined judgment results to the corresponding interaction levels. The instruction number set corresponding to each interaction level is stored using a predefined mapping relationship, with different levels corresponding to different types and numbers of instruction combinations. Before output, the instruction number set is sorted according to priority order, and instructions of the same type are merged to avoid repeated execution and ensure the continuity of the interaction process.

[0139] The IoT-based voice interaction and recognition system for medical companion robots includes:

[0140] The vocal range construction module is used to collect continuous audio streams from the ward, as well as bed pressure, human presence, location tags, and door magnetic data. Based on the bed coordinates and the patient's activity range, it constructs a vocal range model for the target patient.

[0141] The vocal event extraction module is used to perform frame-by-frame processing on the continuous audio stream, extract the spatial features of the sound source and the physiological modulation features of vocalization, and combine the target patient's vocal interval model to filter speech segments and generate real vocal event segments.

[0142] The vital signs synchronization module is used to collect vital signs monitoring data and treatment status data, perform time alignment and identify abrupt changes, construct a vital signs synchronization time window, and map real vocal event fragments to the vital signs synchronization time window.

[0143] The semantic generation module is used to generate a set of disease course semantics based on nursing level, disease label, diagnosis and treatment tasks, and the synchronous time window status of vital signs.

[0144] The speech recognition module is used to extract speech feature sequences from real speech event segments and input them into the improved Paraformer model to perform multi-source semantic-driven feature reconstruction and decoding processing to generate a candidate semantic set.

[0145] The credibility calculation module is used to calculate the credibility of nursing triggers based on real vocal event fragments, candidate semantic sets, and vital sign synchronization time windows.

[0146] The judgment and execution module is used to perform hierarchical judgment processing based on the nursing trigger credibility, candidate semantic set, and the status category corresponding to the vital signs synchronization time window, and generate a set of interactive instructions.

[0147] Example 1: To verify the feasibility of this invention in practice, it was applied to a voice interaction system for a medical companion robot in the neurology ward of a tertiary hospital. This ward is a typical open-plan multi-bed ward with eight beds. Patients are mainly recovering from stroke and suffering from chronic neurological diseases. They generally exhibit unclear pronunciation, low voice intensity, and incomplete expression. Furthermore, the ward is constantly exposed to complex interference from equipment alarms, conversations among medical staff, and other patients' voices. Traditional voice recognition systems in this scenario rely primarily on a single audio signal for recognition, failing to distinguish between the target patient and interfering sound sources. Moreover, they do not incorporate patient vital signs for auxiliary judgment, resulting in a high false trigger rate and a low effective recognition rate, thus failing to meet actual nursing needs.

[0148] In this embodiment, the method of the present invention is deployed in a medical companion robot. The robot collects continuous audio streams from the ward in real time through a microphone array, and simultaneously obtains spatial location information through a bed pressure sensor, a human presence sensor, a positioning tag, and a door magnetic sensor to construct a target patient's vocal range model. During actual operation, when a patient vocalizes within the bed area, the system first filters out audio frames located in the main vocal range or off-center vocal range using a spatial constraint model, and extracts real vocal event segments by combining short-time energy, fundamental frequency continuity, and respiratory modulation features, thereby effectively eliminating interfering speech from other beds or environmental noise. Subsequently, the system synchronously collects vital sign data such as the patient's heart rate, blood oxygen, respiratory rate, and infusion status. By constructing a synchronous time window for vital signs through time alignment and abrupt change detection, the system associates vocal events with the patient's physiological state, so that key vocal expressions such as "calling for help" and "expressing pain" are preferentially identified when there are abnormal changes in vital signs.

[0149] In the speech recognition stage, the system inputs real speech event fragments into an improved Paraformer model. A multi-source semantic driving unit encodes nursing level, disease label, and treatment task information and fuses them with speech features, giving the model medical semantic prior constraints during decoding. Simultaneously, a dual-path decoding structure generates basic text sequences and semantically enhanced text sequences, respectively, and extracts key semantic fragments through a semantic difference mechanism to generate a candidate semantic set. Further, the system calculates nursing trigger credibility based on speech intensity, semantic score, and physiological state score, and performs a tiered judgment based on the state category, outputting corresponding interactive instructions, such as voice broadcasts, nursing reminders, or emergency alarm reports.

[0150] To verify the technical effect of the present invention, the method of the present invention was compared with the traditional speech recognition method (based only on audio signals) in the same ward environment. The test was conducted continuously for 7 days, and a total of 1200 voice interaction events were collected. Key indicators such as recognition accuracy, false trigger rate, response time and emergency event recognition rate were statistically analyzed. The experimental results are shown in Table 1.

[0151] Table 1. Comparison of Voice Interaction Recognition Performance of Medical Companion Robots

[0152] Indicator Name Traditional methods Method of the present invention Total number of voice events (times) 1200 1200 Number of valid speech recognition attempts 872 1096 Recognition accuracy (%) 72.7 91.3 Number of false triggers (times) 198 52 False trigger rate (%) 16.5 4.3 Average response time (ms) 820 610 Success rate of weak speech recognition (%) 58.2 86.7 Non-standard expression recognition rate (%) 61.5 88.9 Number of emergency incidents identified (times) 76 112 Emergency event identification rate (%) 63.3 93.3 Multi-bed interference identification accuracy (%) 68.4 90.1

[0153] As shown in Table 1, this invention significantly improves the accuracy and stability of speech recognition in complex medical scenarios. By introducing a vocalization interval model, the system's interference suppression capability is significantly enhanced in multi-bed environments, reducing the false trigger rate by over 70%. Through the vital sign synchronization time window, the system's ability to recognize abnormal speech events is significantly improved, increasing the emergency event recognition rate by approximately 30%. Through the improved Paraformer model and multi-source semantic driving mechanism, the recognition capability of weak speech and non-standard expressions is significantly enhanced, improving the overall recognition accuracy by nearly 20%. Furthermore, the system response time is also reduced, meeting the requirements of real-time interaction.

[0154] In summary, this invention achieves accurate recognition and intelligent decision processing of target patient speech in complex ward environments through spatial constraints, time synchronization, multi-source semantic fusion, and model structure improvement. It effectively solves the problems of low recognition accuracy, high false trigger rate, and lack of medical semantic support in existing technologies, and has good engineering application value.

[0155] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A voice interaction recognition method for medical companion robots based on the Internet of Things, characterized in that, Includes the following steps: Step 1: Collect continuous audio streams from the ward, as well as data on bed pressure, human presence, location tags, and door magnetic sensors. Construct a target patient's vocal range model based on bed coordinates and the patient's activity range. Step 2: Perform frame segmentation processing on the continuous audio stream, extract the spatial features of the sound source and the physiological modulation features of vocalization, and combine the target patient's vocalization interval model to filter speech segments and generate real vocalization event segments. Step 3: Collect vital sign monitoring data and treatment status data, perform time alignment and identify abrupt change points, construct a vital sign synchronization time window, and map the actual vocalization event fragments to the vital sign synchronization time window; Step 4: Generate a disease course semantic set based on the nursing level, disease label, treatment task, and the synchronous time window status of the vital signs; Step 5: Extract speech feature sequences from the real speech event segments and input them into the improved Paraformer model. Combine the pathological semantic set with the speech feature sequence to perform multi-source semantic-driven feature reconstruction and decoding processing to generate a candidate semantic set. Step Six: Calculate the nursing trigger credibility based on the actual vocal event fragments, candidate semantic set, and vital sign synchronization time window; Step 7: Perform hierarchical judgment processing based on the nursing trigger credibility, candidate semantic set, and the status category corresponding to the vital signs synchronization time window to generate an interactive instruction set.

2. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step one includes: Read the pressure distribution data output by the bed pressure sensor, construct the bed surface pressure matrix according to the preset sampling period, and calculate the pressure concentration area, pressure center location and pressure distribution boundary based on the bed surface pressure matrix; Read the occupancy status data output by the human body presence sensor to determine whether the patient is in bed or out of bed, and read the positioning tag data to determine the patient's real-time spatial coordinates; Read the door magnetic data to determine whether the door is open or closed, and define the doorway activity area based on the door's position coordinates and a preset spatial range; Under a unified ward coordinate system, coordinate alignment is performed on the bed center coordinates, pressure center location, pressure distribution boundary, real-time spatial coordinates of the patient, occupancy status, and doorway activity area. The patient's main distribution area is calculated based on the bed center coordinates and the pressure center location; the posture offset area is calculated based on the pressure distribution boundary and the patient's real-time spatial coordinates; and the external interference area is determined based on the doorway activity area. Under a unified ward coordinate system, the patient's main distribution area, posture deviation area, and external interference area are combined to construct a target patient vocal range model that includes the main vocal range, the deviation vocal range, and the inhibition area.

3. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step two includes: The continuous audio stream is divided into frames according to a preset frame length and frame shift to obtain an audio frame sequence, and windowing processing is performed on each audio frame. Based on the multi-channel audio data collected by the microphone array, cross-correlation calculation is performed on each audio frame to determine the time delay corresponding to the peak value of the cross-correlation function, and the sound source direction angle and sound source distance information are calculated based on the time delay. Short-time energy sequences are calculated for each audio frame. The energy difference between adjacent audio frames is calculated and normalized to obtain the energy change rate. The fundamental frequency trajectory is extracted and the difference change between consecutive fundamental frequency frames is calculated. At the same time, bandpass filtering is performed on the low-frequency envelope to extract the periodic fluctuation amplitude. Under the unified ward coordinate system, the sound source direction angle and sound source distance information are converted into spatial coordinate positions, and it is determined whether the spatial coordinate position falls into the main sound area or the offset sound area corresponding to the target patient's sound interval model. The audio frames that fall within the target patient's vocal range model are counted sequentially in time. Audio frame sequences with a number of consecutive frames greater than the frame count threshold are spliced ​​together to generate real vocal event segments.

4. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step three includes: The vital signs monitoring data and treatment status data are timestamped according to a unified time base to form a multidimensional time series arranged in chronological order. The difference between each parameter in the multidimensional time series at adjacent sampling times is taken as the change amount. The absolute value of the change amount is processed and compared with the change threshold of the corresponding parameter one by one. When the change amount of at least two consecutive sampling times is greater than the corresponding change threshold, the starting sampling time of the continuous sampling time interval is determined as the change point. Using the timestamp corresponding to the mutation point as the center, the time range is extended forward and backward respectively to construct a time window for synchronizing vital signs; The time range of the actual speech event segment is overlapped with the time window of the vital signs synchronization. The ratio of the overlap time length to the duration of the speech segment is calculated, and the actual speech event segment with a ratio greater than the ratio threshold is identified as the synchronized speech segment.

5. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step four includes: Read nursing level information, divide the nursing level into multiple level categories according to the preset level range, and number the semantic words corresponding to each level category to form a basic semantic index set; Read the disease label information, find the corresponding set of symptom descriptive words based on the disease label, and perform vector encoding on each semantic word in the set of symptom descriptive words to form a set of disease semantic vectors. Read the diagnosis and treatment task information, divide the diagnosis and treatment task into multiple task states according to the execution stage, and number the set of operation words corresponding to each task state to form a task semantic index set. The system reads the state of the synchronous time window of vital signs, and determines the rising state of parameters with positive differences, the falling state of parameters with negative differences, and the stable state of parameters with changes less than the stability threshold, based on the sign of the difference between parameters corresponding to the mutation point at adjacent sampling times. The system also encodes the physiological response semantic words corresponding to different states to form a physiological semantic set. The basic semantic index set, disease semantic vector set, task semantic index set, and physiological semantic set are concatenated, and duplicate values ​​are removed according to the semantic number to generate the disease course semantic set.

6. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step five includes: Pre-emphasis, framing, windowing, and Mel spectrum transform are performed on real speech event segments to construct speech feature sequences. The speech feature sequences, disease course semantic sets, and vital signs synchronous time window states are then input into the improved Paraformer model. The improved Paraformer model includes a sequentially connected multi-source semantic driving unit, encoder, predictor, dual-path decoder, and semantic difference unit. The encoder includes a multi-layer stacked feedforward module, self-attention module, and convolution module. The dual-path decoder includes a basic decoding path and a semantic enhancement decoding path. The multi-source semantic driving unit performs numbering, sorting, and vector encoding on the semantic items in the disease course semantic set, constructs a semantic encoding sequence, and maps the state of the vital signs synchronization time window into a state encoding vector. The multi-source semantic driving unit copies the state encoding vector along the time axis to the same length as the speech feature sequence according to the frame order of the speech feature sequence, expands the semantic encoding sequence according to the frame order and maps it to the same length and feature dimension as the speech feature sequence, and performs frame-by-frame splicing, linear transformation and element-wise weighting on the speech feature sequence, state encoding vector and semantic encoding sequence to construct the semantic driving feature sequence. The encoder performs multi-layer temporal encoding on the semantically driven feature sequence. In each layer, the semantically driven feature sequence is input into the self-attention module and the convolution module respectively. The output of the self-attention module and the output of the convolution module are summed and then input into the feedforward module to form the encoded temporal representation. The predictor performs length prediction and position alignment on the encoded temporal representation to construct the basic decoded input sequence; The basic decoding path performs character-level decoding processing on the basic decoding input sequence to construct the basic text sequence; The semantic enhancement decoding path concatenates and performs linear transformation on the semantic encoding sequence corresponding to the pathological semantic set and the basic decoding input sequence according to the position alignment result of the predictor output to construct the semantic enhancement decoding sequence, and performs character-level decoding processing on the semantic enhancement decoding sequence to construct the semantic enhancement text sequence; The semantic difference unit performs position alignment processing on the basic text sequence and the semantically enhanced text sequence, identifies the difference character segments and difference word segments, extracts the newly added semantic segments, the replaced semantic segments and the common semantic segments, and performs filtering and recombination processing according to the semantic numbers in the disease course semantic set to construct a candidate semantic set.

7. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step six includes: Speech intensity features and speech duration features are extracted from real speech event segments. Speech intensity features are calculated by summing the squares of the amplitude values ​​of each audio frame and then performing normalization processing. Speech duration is calculated by multiplying the number of consecutive frames by the frame length. For each semantic item in the candidate semantic set, find the corresponding weight coefficient according to the semantic number, and perform weighted summation on each semantic item according to the semantic category to calculate the semantic score. Normalize the changes of each parameter in the synchronous time window of vital signs, find the corresponding state weight according to the direction and magnitude of change, perform weighted summation on the state weights of each parameter, and calculate the physiological state score. The speech intensity features, speech duration features, semantic score and physiological state score are weighted and summed, and the weights of each item are allocated according to the proportional coefficient to calculate the nursing trigger confidence value. The nursing trigger confidence value is compared with the confidence threshold, and the result of the nursing trigger confidence value being greater than the confidence threshold is determined as a valid trigger event.

8. The voice interaction recognition method for IoT-based medical companion robots according to claim 1, characterized in that, Step seven includes: Read the nursing trigger credibility, candidate semantic set and the status category corresponding to the vital signs synchronization time window, and find the corresponding category label for each semantic item in the candidate semantic set according to the semantic number to construct a semantic category sequence; The credibility of the nursing trigger is compared with the credibility grading threshold range to determine the range in which the nursing trigger credibility is located, and the status category is determined according to the status flag corresponding to the change direction of each parameter in the vital signs synchronization time window. Based on the interval of nursing trigger credibility, semantic category sequence and state category, a combined judgment vector is constructed, and a hierarchical mapping process is performed on the combined judgment vector to determine the interaction level; Each interaction level is pre-configured with a corresponding set of instruction numbers, and each set of instruction numbers includes at least one of voice broadcast instructions, device control instructions, nursing reminder instructions, and alarm reporting instructions. Find the corresponding set of instruction numbers based on the interaction level, and execute and sort the instructions according to their numbers to generate a set of interactive instructions.

9. A voice interaction recognition system for a medical companion robot based on the Internet of Things, comprising executing the voice interaction recognition method for a medical companion robot based on the Internet of Things as described in any one of claims 1 to 8, characterized in that, include: The vocal range construction module is used to collect continuous audio streams from the ward, as well as bed pressure, human presence, location tags, and door magnetic data. Based on the bed coordinates and the patient's activity range, it constructs a vocal range model for the target patient. The vocal event extraction module is used to perform frame-by-frame processing on the continuous audio stream, extract the spatial features of the sound source and the physiological modulation features of vocalization, and combine the target patient's vocal interval model to filter speech segments and generate real vocal event segments. The vital signs synchronization module is used to collect vital signs monitoring data and treatment status data, perform time alignment and identify abrupt changes, construct a vital signs synchronization time window, and map real vocal event fragments to the vital signs synchronization time window. The semantic generation module is used to generate a set of disease course semantics based on nursing level, disease label, diagnosis and treatment tasks, and the synchronous time window status of vital signs. The speech recognition module is used to extract speech feature sequences from real speech event segments and input them into the improved Paraformer model to perform multi-source semantic-driven feature reconstruction and decoding processing to generate a candidate semantic set. The credibility calculation module is used to calculate the credibility of nursing triggers based on real vocal event fragments, candidate semantic sets, and vital sign synchronization time windows. The judgment and execution module is used to perform hierarchical judgment processing based on the nursing trigger credibility, candidate semantic set, and the status category corresponding to the vital signs synchronization time window, and generate a set of interactive instructions.

Images