A service robot behavior recognition method based on multi-modal perception fusion

Abstract

The application relates to the technical field of computer systems based on specific calculation models, and discloses a service robot behavior recognition method based on multi-modal perception fusion, which comprises the following steps: a calculation model for switching between a conventional monitoring mode and an active interrogation mode is established, a fuzzy state with low information quantity in a period is recognized as a specific risk signal, and the system is forcedly switched from passive data analysis to active interrogation based on the specific risk signal, key new evidence is obtained by performing a mild interaction related to a scene, and final behavior recognition is made according to the new evidence; by changing the operation philosophy of the calculation model, the service robot evolves into an intelligent agent capable of actively asking questions when information is unclear, so that the problem that traditional technologies are difficult to deal with dangerous silence is solved.

Description

Technical Field

[0001] This invention relates to a service robot behavior recognition method based on multimodal perception fusion, belonging to the field of computer system technology based on specific computational models. Background Technology

[0002] Currently, a common technical approach is to use computational models to process information streams from multimodal sensors for human behavior recognition. When data containing clear event characteristics is acquired, such as rapid displacement or posture changes in visual information, such computational models can effectively identify specific behaviors. However, in specific application scenarios, such as monitoring elderly users, some states requiring close attention do not produce clear event characteristics but instead manifest as prolonged periods of stillness. In these cases, although the data input streams from each sensor are independent and stable, the lack of specific event information limits the processing flow of computational models that rely on feature data for judgment to continuous analysis of existing information. They are unable to distinguish between two completely different states of stillness, such as normal rest and loss of consciousness.

[0003] Simply increasing the number of sensors or improving the complexity of data fusion algorithms cannot change the fact that the input information itself lacks event characteristics. Therefore, the output results still cannot surpass the confirmation of the ambiguous state of continuous stillness. Specifically, the existing technical methods mainly have the following shortcomings: 1. The operation mechanism of the computational model is limited to passively receiving and analyzing data. When the input information does not have characteristics, it cannot determine its potential risk attributes; 2. The system's processing flow is unidirectional. When the current information is insufficient to support a clear judgment, there is no corresponding mechanism to actively acquire new and distinguishable information. This computational paradigm, which is limited to passive data processing, is common even in some robot control methods centered on dynamic task scheduling. In essence, it still fails to solve the decision-making dilemma when facing cognitive uncertainty.

[0004] For example, Chinese invention patent CN120669580A discloses a control method and system for catering service robots based on multimodal perception. Although this method utilizes multimodal perception technologies such as RFID and vision to identify the type of dishes and user level, and then dynamically adjusts task priority and path planning to achieve efficient scheduling optimization in catering scenarios, its fundamental starting point for control logic is to execute a well-defined external task (such as delivering hot drinks to VIP users). The intelligence of this system is only reflected in how to complete a known task faster and better, and its entire operation process is still a passive response to external instructions. When faced with scenarios such as the aforementioned dangerous silence, which have no clear task instructions and are full of uncertainty, this model lacks the necessary internal driving mechanism to actively explore and define problems. It cannot generate an internal task oriented towards eliminating uncertainty (such as actively inquiring about the user's status) from a continuous ambiguous state, and it has an application bottleneck in monitoring scenarios that require active cognitive reasoning. Therefore, the technical problem to be solved by this invention is how to design a computational model that can identify the hidden uncertainties in continuous featureless sensor information and proactively change its workflow to obtain new information from the external environment that can be used for final judgment. Summary of the Invention

[0005] This invention provides a service robot behavior recognition method based on multimodal perception fusion. Its main purpose is to solve the problem that existing computing models, due to their passive analysis processing flow, cannot make clear behavior judgments when faced with continuous input information without features.

[0006] To achieve the above objectives, this invention provides a service robot behavior recognition method based on multimodal perception fusion, which includes the following steps:

[0007] Step a: The service robot is operated in a regular monitoring mode. In this mode, it continuously receives sensor data from at least two modes and sets a first threshold and a time window. Then, the calculation rule for the cognitive uncertainty index is executed. The calculation rule is as follows: when no mode can independently output a concern behavior label with a confidence level higher than the first threshold within the time window, and at least one mode continuously outputs concern behavior labels representing a neutral state, a cognitive uncertainty index increases cumulatively over time.

[0008] Step b: Set a second threshold and establish a mode switching rule. The mode switching rule is: if and only if the cognitive uncertainty index exceeds the second threshold, switch the operation mode of the method from the regular monitoring mode to the active questioning mode.

[0009] Step c: In the active questioning mode, pause the fusion analysis of sensor data and control the service robot to perform non-command-based, context-dependent voice interaction to obtain response information;

[0010] Step d: Based on the response information, determine the final behavior recognition result, and switch the operation mode of the method back to the conventional monitoring mode according to the mode switching rules.

[0011] Preferably, in step c, when the response information is a speech response, the step d in which the final behavior recognition result is determined based on the response information further includes: performing dual-path parallel parsing on the speech response, wherein the first parsing path is used to determine the linguistic content of the speech response, and the second parsing path is used to extract a set of paralinguistic features from the original audio waveform of the speech response; comparing the extracted set of paralinguistic features with the user's personal normal value baseline to obtain the deviation, the user's personal normal value baseline being established based on the user's speech data continuously collected and updated in daily interactions; and collaboratively determining the final behavior recognition result based on the linguistic content and the deviation.

[0012] Preferably, before performing dual-path parallel parsing on the speech response, the method further includes: using a visual sensor to determine the three-dimensional spatial position of the user's mouth in the service robot's own coordinate system; based on the three-dimensional spatial position, controlling the microphone array to perform an acoustic beamforming algorithm to form an auditory focus pointing to the three-dimensional spatial position, so as to amplify the sound signal from the target direction and suppress environmental noise from other directions when acquiring the speech response, and using the audio signal generated after this processing as the speech response for dual-path parallel parsing.

[0013] Preferably, the method further includes: under the conventional monitoring modality, detecting in real time whether there are at least two modalities whose output behavioral information constitutes a conflict in a predefined semantic conflict matrix, and whose confidence levels are all higher than the conflict confidence threshold; if a conflict is detected, bypassing the calculation of the cognitive uncertainty index, directly switching the operation modality of the method from the conventional monitoring modality to the active questioning modality.

[0014] Preferably, the method further includes: establishing a personalized circadian rhythm baseline for the user based on long-term accumulated, timestamped sensor data under conventional monitoring modes, which depicts the user's typical behavioral patterns at different times of the day; in step b, the second threshold is dynamically adjusted according to the current time point and the personalized circadian rhythm baseline, and the adjustment rule is: the second threshold is increased during the time period when the user is usually in a static state according to the personalized circadian rhythm baseline, and decreased during the time period when the user is usually in an active state.

[0015] Preferably, the set of paralinguistic features extracted in the second parsing path includes at least one of the following: the vocalization delay time from the end of the robot's question to the start of the user's response, the fundamental frequency perturbation characterizing the stability of vocal cord vibration, and the intonation flatness characterizing the richness of intonation variation.

[0016] Preferably, in step c, if no response information is obtained after performing non-instructional, context-related voice interaction, the service robot is further controlled to perform at least one of the following actions: performing environmental stimuli aimed at attracting the target's attention, or changing the physical observation perspective of the sensors.

[0017] Preferably, in step d, when the response information indicates that there is no effective response to the proactive questioning action, the final behavior identification result is determined to be a high-risk state, and an external alarm is triggered.

[0018] Preferably, in the step of dynamically adjusting the second threshold, the adjusted threshold Determined by the following rules: ,in, Based on the threshold, The modulation factor is determined based on the current time. With the determination of the personalized rhythm baseline, if the current time If it falls within the resting period defined by the personalized rhythm baseline, then If the current time is positive, If it falls within the active period defined by the personalized rhythm baseline, then It is a negative value.

[0019] Preferably, in the step of collaboratively determining the final behavior recognition result, the following collaborative judgment rules are established: when the linguistic content is clear and the deviation is below the deviation threshold, it is determined to be a safe state, and the modality switching rule is executed; when the linguistic content is clear but the deviation is above the deviation threshold, it is determined to be an abnormal state of sound features, and a specific alarm is triggered; when the linguistic content is unclear and the deviation is above the deviation threshold, it is determined to be a highly suspected medical emergency, and the highest level alarm is triggered.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] 1. In routine monitoring, the system continuously outputs preliminary behavioral judgments based on multimodal sensor information. When the system continuously obtains neutral or low-information behavioral labels within a specific time period but fails to output a high-confidence specific behavioral conclusion, an index representing the uncertainty of the system's cognition accumulates. Once the index reaches a preset state, it triggers the entire computing model to switch from passive information fusion analysis to an active questioning process. This process controls a service robot to perform a preset physical action to obtain new response information in the environment and determines the final behavioral recognition result based on this response information. This structure, which switches between two different types of processes driven by a continuous fuzzy state, enables the computer system to proactively break out of the passive analysis dilemma and solve problems with the aim of obtaining new evidence when facing scenarios that are difficult for traditional fusion algorithms to handle, such as those with insufficient or unclear information.

[0022] 2. After acquiring the user's voice response during the proactive questioning process, a dual-path parallel voice information parsing structure was established. The raw audio stream captured by the robot microphone was simultaneously fed into two processing channels. The first channel used speech recognition technology to parse its linguistic content to determine the literal meaning of the response. The second channel used signal processing algorithms to directly extract a set of paralinguistic features that could reflect the speaker's physiological state from the raw audio waveform. These real-time extracted features were compared with the user's personal normal baseline stored in the system to obtain a deviation. The final behavior recognition result was a judgment made based on the parsing result of the linguistic content and the deviation of the paralinguistic features. This judgment method, which orthogonally parses the same signal source and compares it with a personalized baseline, allows the system's assessment of the user's state to no longer rely solely on the understanding of the meaning of language, but to extend to the insight into the physiological changes behind the vocal behavior.

[0023] 3. Before performing the above-mentioned dual-path parallel speech information parsing, a cross-modal perception information guidance mechanism was also established. The system first uses a visual sensor to recognize the face and determine the three-dimensional spatial position of the user's mouth in the robot's own coordinate system. Then, the computational model uses this spatial position information as a target parameter and passes it to the microphone array signal processor on the robot. The processor then executes an acoustic beamforming algorithm to form a narrow-domain auditory focus pointing to this spatial position at the software level. This is used to amplify the sound signal from the target direction and suppress environmental noise from other directions when collecting the speech response. The audio signal with a higher signal-to-noise ratio generated after this processing is then sent to the subsequent dual-path parallel parsing structure. This processing flow, which uses the visual positioning result to accurately guide the acoustic signal acquisition focus, enables the system to actively optimize its auditory perception conditions before preparing for refined speech feature analysis.

[0024] 4. This method also establishes an adaptive adjustment mechanism for the cognitive uncertainty index threshold based on the user's long-term behavioral rhythm. Under normal monitoring operation, the system continuously accumulates and stores preliminary behavioral information by timestamp, and establishes a personalized rhythm baseline for a specific user by performing time series analysis in the background, which describes the typical behavioral patterns of the user at different times of the day. When performing behavior recognition, the preset threshold of the cognitive uncertainty index used to trigger the switching of work mode will be dynamically adjusted according to the current time point and the personalized behavioral rhythm baseline. For example, the threshold will be increased during the time period when the user is usually in a static state, and decreased during the time period when the user is usually active. This method of adjusting the sensitivity of real-time judgment with the individual behavioral patterns established by long-term observation makes the triggering conditions for the switching of model operation mode able to fit the lifestyle habits of different users. Attached Figure Description

[0025] Figure 1 This is a flowchart of the cognitive uncertainty-driven modality switching and behavior recognition process of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the correlation between the Z-score of the secondary linguistic feature of this invention and the abnormal state of the user;

[0027] Figure 3 This is a timing diagram for the fast bypass triggering of high-confidence data conflicts in this invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The service robot behavior recognition method based on multimodal perception fusion claimed in this invention is based on the establishment of a computational model that can switch between a conventional monitoring mode and an active questioning mode. This computational model uses the accumulation of a cognitive uncertainty index to characterize the degree of confusion of the system when facing a persistently low-information ambiguous state. This index serves as a driving force to switch the computational model's operation from passive data analysis to an active interaction process. In the active interaction process, new key evidence is obtained through specific interactions with the environment, and a final behavior judgment is made based on this new evidence. The method also includes a mechanism for dual-path parallel parsing of the voice response obtained after the interaction, and a fast bypass triggering mechanism to deal with high-confidence data conflicts. At the same time, its core judgment threshold can be dynamically and adaptively adjusted according to the user's personalized behavioral rhythm.

[0030] In a typical application scenario, such as monitoring elderly users, existing technologies, when dealing with prolonged periods of inactivity, fail to effectively distinguish between normal rest and loss of consciousness due to the lack of specific event characteristics in the input information. To address this technical problem, the method claimed in this invention is configured to perform the following steps on a service robot: First, the service robot is operated in a conventional monitoring mode, in which the robot's processor continuously receives sensor data from at least two modes, for example, a visual sensor acquires image data at a frequency of 10 Hz, while a millimeter-wave radar acquires the target's respiratory data at a frequency of 5 Hz; the system internally sets a first threshold for judging high-confidence behavior. With a time window This parameter is determined through a standardized data-driven optimization procedure, targeting the first threshold. The determination process includes, for example, using a manually labeled multimodal sensor dataset that contains a sufficient number of event segments clearly belonging to the behavioral labels of interest, such as falls and loss of consciousness, as well as a large number of event segments representing neutral states, such as sitting still and sleeping; The value is used as the scanning variable, and it is traversed within the interval [0,1]. For each candidate... The value is calculated across the entire dataset, determining the true positive rate (TPR) for correctly identifying the behavior label of concern and the false positive rate (FPR) for incorrectly identifying neutral states as requiring attention; all alternatives are considered. The FPR and TPR values ​​are plotted as receiver operating characteristic (ROC) curves, and the value corresponding to the inflection point on the curve that is closest to the ideal point (0,1) in Euclidean distance is automatically selected as the optimal first threshold that balances recognition sensitivity and accuracy. Regarding the time window The determination procedure includes: based on the previously determined optimal first threshold The test is conducted on a dataset containing various normal and abnormal behavioral transition scenarios. Using a value of T, for example, ranging from 10 seconds to 180 seconds, the scan variable is iterated. For each candidate T value, the average alarm delay time in dangerous, silent scenarios such as simulated loss of consciousness, and the frequency of triggering unnecessary interrogations in normal, long-duration still scenarios such as naps, are evaluated. Finally, a T value is selected that ensures the alarm delay time is below the safety limit when a real risk occurs, while also controlling the frequency of disturbance to the user's normal behavior within an acceptable range. In a specific embodiment, the first threshold P determined by this procedure can be 0.85, the time window T can be 60 seconds, and a compromise value can be set to 60 seconds. Then, the processor executes a calculation rule for a cognitive uncertainty index, which is defined as: when the length is... Within the time window, no modality can independently output a value with a confidence level higher than the first threshold. Furthermore, the behavior tags belong to the preset list of behaviors to be monitored (e.g., falling, abnormal posture), and when at least one modality continuously outputs a tag representing a neutral state (e.g., still or sitting posture), an index with an initial value of 0 is used to represent cognitive uncertainty. It increases cumulatively over time; as a numerical example, if for 60 consecutive seconds, the visual sensor continuously outputs (still, 0.7), and the radar sensor continuously outputs (stable breathing, 0.9), since no attention behavior label has a confidence level exceeding 0.85, and the neutral label of "still" exists, then... The index will start from 0 and accumulate at a rate of 1 unit per second, ending at the end of 60 seconds. The value will reach 60. In this way, the system continuously quantifies the state of ambiguity into a specific risk indicator parameter. In addition, the calculation rule of the cognitive uncertainty index is reflected in the scenario that can also handle the loss or anomaly of some sensor data. If the visual sensor continuously outputs (stationary, 0.7) for 60 consecutive seconds, but the millimeter-wave radar fails for some reason, causing its data stream to be interrupted or outputting invalid readings, the accumulation condition of the cognitive uncertainty index CUI still holds. This is because it still meets the two core conditions that at least one modal visual sensor continuously outputs a label representing a neutral state, and no modality can independently output a label of attention behavior with a confidence level higher than the first threshold P. Therefore, the CUI index will still start from 0 and accumulate at a rate of 1 unit per second. This design ensures that when the perceived information is incomplete, the system can also identify the uncertainty caused by sensor failure as a potential risk that needs attention and drive subsequent modality switching, rather than falling into a state of inaction due to missing data, thereby enhancing the reliability of the entire system in real complex environments.

[0031] Therefore, in order for the system to respond to this quantified risk indicator parameter, the system sets a second threshold to trigger a switch in operating mode. And establish mode switching rules; second threshold The setting is based on the acceptable upper limit of risk accumulation. For example, if the second threshold is set to 90, it means that the system allows users to remain in a static state that cannot be understood with high confidence for a maximum of 90 seconds. Exceeding this limit is considered to indicate that the risk level has reached a level requiring active intervention. The base value of the second threshold used to trigger modality switching is... The Cognitive Uncertainty Index (CUI) is not an empirical value, but rather determined through a standardized data-driven optimization procedure. This procedure first utilizes a manually labeled multimodal sensor dataset covering long-term user behavior. A CUI calculation module is run on this dataset to generate a continuous time series curve of CUI values. Before executing the CUI calculation module, the procedure includes a data preprocessing and anomaly handling step. This anomaly handling step addresses potential interruptions or missing segments of sensor data in the dataset. Specifically, a missing duration threshold is set, for example, 2 seconds. If the missing duration is less than this threshold, data is supplemented using linear interpolation or previous value imputation to maintain the continuity of the sequence. If the missing duration exceeds this threshold, a valid state interruption is considered to have occurred, and the CUI value at that time point is forcibly reset to 0, with the accumulation calculation restarting from the next moment after data recovery. This anomaly handling mechanism ensures that long periods of data loss are not incorrectly accumulated into a high CUI, avoiding false alarms. Therefore, this procedure... The values ​​are used as scanning variables, traversing within a preset interval. For each candidate threshold, the false negative rate (the proportion of all labeled scenarios where a challenge should be triggered that failed to do so) and the false positive rate (the proportion of all normal long-term static scenarios where an unnecessary challenge was triggered) are calculated across the entire dataset. Finally, the procedure plots the false positive rate and the corresponding true positive rate (1 minus the false negative rate) as a receiver operating characteristic (ROC) curve, and automatically selects the value corresponding to the inflection point on the curve that is closest to the Euclidean distance from the coordinate point (0,1) as the final value, balancing response sensitivity and user intrusion. The established modality switching rule is: the cognitive uncertainty index is determined if and only if the cognitive uncertainty index is within a certain range. The value exceeds the second threshold At this time, the operating mode of the computational model switches from the conventional monitoring mode to the active questioning mode; the essence of this switching action is the active state transformation of the computational model, which changes the focus of the system's work from understanding existing data to acquiring new data.

[0032] Upon entering the active questioning mode, to avoid being continuously interfered with by existing ambiguous information, the system processor suspends the fusion analysis of data from various modal sensors. The goal is to acquire new information that can break the current cognitive deadlock with minimal interaction cost. Based on a pre-set questioning toolbox, the system controls the service robot to perform a non-command, context-dependent voice interaction. For example, if the system's internal clock shows 6 PM, the robot will use its speech synthesis unit to ask in a pre-set tone: "It seems dark outside; should I turn on the living room lights?" It should be noted that to ensure the effectiveness of the interaction, the interaction path here is designed as multi-channel, with voice interaction being the preferred path. If no response is received from the user within a pre-set response waiting time (e.g., 10 seconds), the system automatically switches to a backup interaction path, controlling the service robot to perform an environmental stimulus aimed at attracting the target's attention, such as increasing the robot's screen brightness by 50% within 2 seconds. This multi-channel interaction... The design of the interactive scheme ensures that even if the user's hearing is poor or the environment is noisy, obstructing the preferred voice path, the system can still continue to execute the challenge task through the backup path, thus eliminating the practical obstacle of method failure due to the failure of a single interaction path. After executing the challenge action, the system analyzes the response information triggered by the challenge and determines the final behavior recognition result based on this response information. If the response information is a logical response from the user to the voice question, such as the voice recognition module recognizing the user's answer as "yes," the system obtains information that the user is conscious and determines the final behavior recognition result as a safe state. If all active challenges, including voice interaction and environmental stimuli, fail to obtain any effective response that can be captured by the sensors, the system takes this challenge failure as a risk signal, determines the final behavior recognition result as a high-risk state, and immediately triggers an external alarm through the network interface. After determining the final result, the system, according to the modality switching rules, adjusts the cognitive uncertainty index. Reset to zero and switch the method's operating mode from active questioning mode back to regular monitoring mode.

[0033] To further improve the precision of user status assessment, the steps for determining the final behavior recognition result based on the response information are further refined when the user's response is a voice response. Given that some users' emergency states may manifest as the ability to speak but unclear speech, the system performs dual-path parallel parsing upon receiving the raw audio stream of the voice response. In the first parsing path, an embedded speech recognition engine processes the audio to determine its linguistic content. Simultaneously, the second parsing path uses signal processing algorithms to directly extract a set of preset paralinguistic features from the raw audio waveform. These features include phonation delay time, fundamental frequency perturbation representing the stability of vocal cord vibration, and intonation flatness representing the richness of intonation variation. Subsequently, the system compares this set of real-time extracted paralinguistic features with a personal normal baseline for the user to obtain the deviation. This personal normal baseline is established and maintained through continuous unsupervised learning of daily user conversational speech data collected under regular monitoring modalities during the system's daily operation. The final decision is made by… Linguistic content and paralinguistic deviation are determined collaboratively, and their judgment rules are solidified into a decision matrix: when the linguistic content is clear and the deviation is below the preset deviation threshold, it is determined to be a safe state; when the linguistic content is clear but the deviation is above the threshold, it is determined to be an abnormal voice feature state, and a graded alarm response protocol is initiated. This protocol clarifies the specific transition measures and subsequent processing procedures under different risk levels: Determined to be a safe state: when the linguistic content is clear and the deviation is below the preset deviation threshold, the system determines that the user's state is safe, clears the cognitive uncertainty index (CUI) to zero, and switches back to the normal monitoring mode; determined to be an abnormal voice feature state, a level one monitoring alarm is initiated. Transition measures: when the linguistic content is clear but the deviation is above the threshold, the system does not immediately trigger an external alarm, but instead initiates a transitional monitoring procedure. This procedure controls the robot to initiate a clarifying dialogue for the purpose of confirming the user's well-being, such as asking: I heard what you said clearly, but your voice feels a little different than usual. Are you feeling okay now? The system will reassess the state based on the user's response to this secondary inquiry or no response.If the user confirms there are no issues, the system returns to regular monitoring. If the response remains abnormal or nonexistent, the alarm level is escalated to Level 2, indicating a highly suspected medical emergency. Level 2 emergency alarm tiered processing and subsequent actions are initiated: When linguistic content is unclear and the deviation exceeds a threshold, or when escalated from a Level 1 alarm, the system determines it as a high-risk emergency and immediately activates the highest-level emergency alarm procedure. This procedure includes one or more of the following actions: Executing an external alarm: Immediately sending an alarm notification via the network module to pre-defined emergency contacts such as children, community service center mobile applications, or telephones; Sending an event data packet: Packaging and sending key evidence triggering the alarm, including: the reason for the interrogation, such as prolonged silence, fragments of the user's abnormal voice response, and a brief analysis report of the deviation of paralinguistic features, providing a basis for the recipient's judgment; Entering on-site support mode: The robot automatically executes a series of pre-defined on-site support actions, such as: streaming its own camera footage to emergency contacts in real time, maximizing microphone volume to establish a remote voice channel, and navigating to the door to guide rescue personnel, etc.

[0034] Considering that background noise can severely contaminate the speech signal used for detailed analysis in a real home environment, to ensure the accuracy of the aforementioned paralinguistic feature extraction, the method also includes a cross-modal perceptual information guidance mechanism before performing dual-path parallel parsing on the speech response. Specifically, the system first uses its visual sensor to determine the three-dimensional spatial position of the user's mouth in the service robot's own coordinate system through a face recognition and localization algorithm. Subsequently, the computational model uses this spatial location information As a target parameter, it is passed to the microphone array signal processor on the robot, which then executes an acoustic beamforming algorithm. This algorithm utilizes the time difference of signals received by multiple microphones to form an auditory focus pointing to the target three-dimensional spatial location at the software level. This is used to amplify the sound signal from the target direction and suppress background noise from other directions when acquiring the voice response. After this processing, an audio signal with a higher signal-to-noise ratio is generated, which is then used as input to the subsequent dual-path parallel parsing process. This improves the signal quality from the source of information acquisition. To address the actual situation where the user's face is obscured or the lighting conditions are poor, leading to visual positioning failure, this method also establishes a hierarchical auditory focus determination backoff mechanism: 1. The primary option is visual-guided beamforming. The system defaults to prioritizing the use of visual sensors to determine the three-dimensional spatial position of the user's mouth and guide the microphone accordingly. 1. The primary option is acoustic beamforming; 2. The alternative option is sound source localization beamforming. If visual localization fails to return valid target coordinates within a preset time, such as 1 second, the system will automatically and seamlessly switch to the alternative option. In this option, the microphone array processor will no longer wait for visual input, but will instead execute the sound source localization algorithm. By analyzing the time difference of the user's sound signal received by each microphone, it will autonomously calculate the approximate azimuth of the sound source, i.e., the user, and form an auditory focus pointing in that direction based on this; 3. The final option is omnidirectional sound pickup and post-processing noise reduction. In a few cases, such as when the user's voice is weak and the background noise is extremely high, the sound source localization algorithm may not be able to lock onto a stable sound source. In this case, the system will adopt the final option. In this option, the microphone array will pick up sound in omnidirectional mode to ensure that no signal is missed, and a general digital signal processing noise reduction algorithm will be performed on the acquired mixed signal.

[0035] Furthermore, the method claimed in this invention also establishes a risk perception mechanism for dealing with rapid changes and uncertainties. In real-world scenarios, there may be conflicts between high-confidence conclusions from different sensors. For example, a wearable wristband might report a high heart rate in a standing posture (confidence 0.95), while a camera might identify the user as being in bed and asleep (confidence 0.98). In this case, conventional... The index does not accumulate rapidly due to the presence of high-confidence labels, which may lead to decision-making delays. To address this, the method also includes a parallel bypass triggering logic: In the normal monitoring mode, the system detects in real time at each time step whether there are at least two modes whose output behavioral information constitutes a conflict in a predefined semantic conflict matrix. For example, the conflict value (sleep posture, standing heart rate) is pre-labeled as 1.0 in this matrix. If the system detects a pair of conflicting labels, and their respective confidence levels are both higher than a preset conflict confidence threshold (e.g., 0.9), the system will bypass the calculation of the cognitive uncertainty index and directly switch the method's operating mode from the normal monitoring mode to the active questioning mode. This mechanism adds a high-priority processing path to the computational model.

[0036] Finally, to better adapt the entire method to the individual lifestyles of different users and avoid false alarms or missed alarms caused by fixed threshold settings, this invention also establishes an adaptive adjustment mechanism for the cognitive uncertainty index threshold. Under normal monitoring conditions, the system continuously accumulates and stores the preliminary behavioral information output by each modality in the background by timestamp, and establishes a personalized rhythmic baseline for a specific user, depicting their typical behavioral patterns at different times of the day, through time series analysis algorithms. During real-time behavior recognition, the aforementioned second threshold used to trigger modality switching... Instead of being a fixed value, it is dynamically adjusted based on the current time point and the personalized rhythm baseline. The adjustment rule is defined as follows: during periods when the user is typically in a static state according to their personalized rhythm baseline, the second threshold is increased; conversely, during periods when the user is typically active, the second threshold is decreased. This adjustment is achieved through an adjustment factor, and the adjusted threshold... By rules To determine; among them, For dynamically adjusted time points The final threshold; The base threshold is the unadjusted threshold, for example, 90; To be based on the current time Modulation factors determined by the personalized rhythm baseline, if the current time If it falls within the stationary period defined by the baseline, then A positive value, such as +0.5, raises the threshold to 135. If it falls within an active period, then... A negative value, such as -0.3, lowers the threshold to 63. This adjustment method allows the model's judgment sensitivity to match the user's individual behavior pattern.

[0037] Example 1: In a monitoring scenario for an elderly user with mild cognitive impairment living alone, the user sits quietly on the living room sofa for an extended period in the afternoon. A service robot is deployed in the living room and continuously operates in a routine monitoring modality. The visual sensor outputs a behavioral label of "still" with a confidence level of 0.7. The non-contact physiological sensor detects a stable respiratory rate with a confidence level of 0.9. Because the system fails to obtain any attentive behavioral label with a confidence level higher than 0.85 for more than 90 consecutive seconds, and continuously obtains the neutral label of "still," the index used by the system processor to characterize cognitive uncertainty... The accumulated threshold of 90 is exceeded; at this point, the dynamic adaptive threshold adjustment mechanism based on the user's personalized rhythm baseline is activated. The user's personalized behavioral rhythm baseline stored in the system background shows that the user usually has brief activities in the afternoon, and the current time period is not their typical quiet period. Therefore, the modulation factor... It is calculated as a negative value of -0.3, making the second threshold actually effective. The value was dynamically lowered to 63 due to accumulated [data / effects]. Once the index exceeds the adjusted threshold, the computational model is immediately switched from the conventional monitoring mode to the active questioning mode. This method, which correlates the sensitivity of real-time judgment with the individual behavior patterns established through long-term observation, achieves a balance between ensuring response speed and avoiding excessive interference.

[0038] Upon entering the active interrogation mode, the system pauses the fusion analysis of existing sensor data and executes non-command-based voice interaction. The robot asks in a preset tone: "It seems dark outside, should I turn on the living room light?" The user responds verbally, but the voice is unclear and the content cannot be clearly identified by the linguistic content parsing path. At this point, to address the risk of insufficient voice interaction information acquisition, a mechanism for dual-path parallel parsing of the voice response is activated. Before initiating this parsing process, a cross-modal perception information guidance mechanism operates first. The system uses a visual sensor to determine the three-dimensional spatial position of the user's mouth in the robot's coordinate system and transmits this spatial position information as a target parameter to the microphone array signal processor. The processor then executes an acoustic beamforming algorithm to form an auditory focus pointing to this spatial position. This action uses the visual positioning result to guide the acquisition of acoustic signals, providing a basis for subsequent refined voice characteristics analysis. The feature analysis provides an audio signal with a higher signal-to-noise ratio. Based on the obtained audio signal, the second analytical path extracts a set of paralinguistic features from the original audio waveform. The calculation results show that the user's vocal delay time is 150% longer than their personal normal baseline, and the fundamental frequency perturbation, which characterizes the stability of vocal cord vibration, exceeds the baseline standard deviation by more than three times. The collaborative decision-making mechanism, based on the information input from the two dimensions of unclear linguistic content and high deviation of paralinguistic features, determines the final behavior recognition result as a highly suspected medical emergency and immediately triggers the highest level of external alarm. At the same time, the entire process of questioning-fuzzy response-feature anomaly is recorded and sent as event evidence. The processing flow of this calculation model, by identifying the continuous fuzzy state as a risk signal and actively acquiring new evidence, enables the system to obtain new information of decisive value through active physical interaction when information is insufficient, thereby making a final judgment.

[0039] Example 2: To verify the performance of the method claimed in this invention in distinguishing between long-term static states of different natures, the following comparative experiment was conducted. The purpose of the experiment was to quantify the recognition accuracy and response time of this method when facing two typical scenarios—normal rest and dangerous silence—which are similar in their performance at the sensor raw data level but have different risk levels. The experiment was conducted on an experimental platform simulating a home environment, which consisted of a multimodal data playback system and two parallel computational model processing units. The data source was a pre-recorded dataset containing synchronized visual, audio, and millimeter-wave radar data streams. This dataset was generated by testers simulating various typical behavioral scenarios. The data playback system simultaneously input the same set of data streams into the two processing units. The control group ran a prior art computational model based on post-fusion, which weighted the preliminary recognition results of each independent modality to obtain the final conclusion. The experimental group ran a computational model using the method claimed in this invention, which has the ability to switch between conventional monitoring and active interrogation modalities. Its internal key parameters were set as follows: First threshold Set to 0.85, time window Set to 60 seconds, the base value for adjusting the second threshold. Set to 90.

[0040] The experiment selected three representative test scenarios, with each scenario repeated 10 times. The judgment results and response times of the two calculation models were recorded. Scenario 1 was normal rest, where the test subject sat quietly on the sofa and read for 10 minutes. Scenario 2 was dangerous silence with loss of consciousness, where the test subject sat quietly for 2 minutes and then simulated losing consciousness, with no significant change in body posture but no response to the outside world. Scenario 3 was dangerous silence with slurred speech, where the test subject also simulated a sudden illness after sitting quietly for 2 minutes. When the experimental group initiated a voice inquiry, the test subject was able to make indistinct sounds as a response, but the content was incoherent. The comparison data of the judgment results and response times of the two calculation models are shown in Table 1.

[0041] Table 1: Performance comparison of the two computing models in different scenarios.

[0042] Test Scenario Computational model Has the active challenge mode been triggered? Paralinguistic deviation Final judgment result Alarm delay (seconds) Normal rest control group no not applicable still No alarm Normal rest experimental group no not applicable still No alarm Loss of consciousness control group no not applicable still >600 (Not triggered) Loss of consciousness experimental group yes not applicable High-risk status 95.2 Speech unclear control group no not applicable still >600 (Not triggered) Speech unclear experimental group yes high Highly suspected medical emergency 108.6

[0043] According to the data recorded in Table 1, in Scenario 1, both models correctly determined that the user was in a normal static state and did not generate false alarms. In Scenario 2, the control group, due to its passive fusion calculation method lacking a mechanism to identify continuous low-information data as risk, failed to trigger alarms, resulting in missed alarms. Meanwhile, the experimental group's cognitive uncertainty index... The threshold was exceeded at approximately 90 seconds, triggering an active challenge mode. After receiving no response, a high-risk alert was issued at an average of 95.2 seconds. In scenario three, the experimental group also obtained the user's voice response after triggering active challenge. The second parsing path in its dual-path parallel parsing mechanism detected that the paralinguistic features of the voice deviated from the user's personal normal baseline. Therefore, the collaborative decision-making mechanism judged the state as a highly suspected medical emergency, which was more urgent than no response, and issued the highest level alert. The experimental data shows that the method claimed in this invention can distinguish between benign and malignant static states that are difficult to distinguish at the sensor data level. Through the switching of working modes driven by cognitive uncertainty and the multi-dimensional analysis of response information, timely and accurate judgment can be made when potential risks occur, solving the risk of underreporting in existing passive fusion models when facing data without event features.

[0044] Example 3: This example combines Figures 1 to 3 This section describes a service robot behavior recognition method based on multimodal perception fusion, such as... Figure 1 As shown, the process begins with processing multimodal sensor data input from sources such as vision, millimeter-wave radar, and wearable devices. It first enters the conventional monitoring mode, where the system continuously makes preliminary behavioral judgments and calculates a cognitive uncertainty index in parallel. Meanwhile, a high-confidence data conflict fast bypass triggering mechanism detects semantic conflicts between sensors in real time. Under normal monitoring modes, If the accumulated value exceeds a dynamic adjustment threshold, or if the bypass mechanism detects a conflict, the process mode is switched to active challenge mode. In this mode, the system suspends passive data analysis and actively performs hierarchical challenge actions, such as voice interaction, environmental stimuli, or perspective changes, to obtain new response information. After obtaining the response information, the system performs in-depth analysis of the speech and other signals, specifically through a dual-path parallel parsing mechanism. The first path of this mechanism parses the linguistic content of the speech, while the second path calculates paralinguistic features such as phonation delay and fundamental frequency perturbation, and then compares these features with a statistical model that stores the individual user's speech features. The system compares the user's personal baseline data with the deviation from the target's normal values ​​to obtain the deviation. Then, a collaborative judgment and decision-making module integrates linguistic content and paralinguistic deviation to determine the final result. This final behavior recognition result can be output as a safe status, abnormal voice features, or a high-risk warning. After outputting the result, the system will reset. It then switches back to the regular monitoring modality. Furthermore, the figure also shows an adaptive adjustment mechanism for the cognitive uncertainty index threshold, based on the user's personalized circadian rhythm baseline. The determination of whether the limit is exceeded provides a dynamically adjusted threshold.

[0045] like Figure 2 As shown in the figure, the horizontal axis represents four state types: normal, slightly abnormal, moderately abnormal, and severely abnormal. The vertical axis represents the Z-score, which quantifies the degree of deviation. The figure uses three different styles of bars to represent the Z-score values ​​of three paralinguistic features: vocal delay time, fundamental frequency perturbation, and intonation flatness. It can be clearly observed from the figure that as the state type evolves from normal to severely abnormal, the Z-scores of all three paralinguistic features show an increasing trend, indicating that the deviation of the user's vocal characteristics from their personal normal baseline increases with the severity of the abnormality.

[0046] like Figure 3 As shown, the process begins with conflict detection in routine monitoring. The visual sensor inputs sleep posture data (confidence 0.98) to the processor, while the health wristband simultaneously inputs standing high heart rate data (confidence 0.95). The processor then queries for semantic conflicts in the tags and sends this conflict information to the conflict matrix. After judgment, the conflict matrix returns a conflict value of 1.0 (severe conflict). After confirming that the confidence of both exceeds the threshold, the processor bypasses uncertainty calculation, directly switches to the active challenge mode, and initiates hierarchical challenge. The robot first executes the first level: voice inquiry. If there is no response from the user, it further executes the second level: environmental stimulus (adjusting the bedside lamp). This stimulus triggers a physical response from the user, which is detected by the visual sensor. Finally, the processor determines that it is safe, records the sensor malfunction, and switches back to the routine monitoring mode.

[0047] Example 4: This example aims to provide a detailed description of the establishment, maintenance, and application procedures for user-specific normal value baselines used in dual-path parallel parsing of speech responses. Given the individual differences in vocal characteristics among different users, which may change slowly over time, a fixed, universally applicable paralinguistic feature model lacks robustness in engineering applications. Therefore, the method claimed in this invention employs a dynamic, personalized baseline modeling and application process. When the system is initially deployed in a new user environment, its user-specific normal value baselines are initialized to empty. In an initial phase, set for 14 days, to address the core issue of the inability to calculate deviations due to a lack of personal baseline data, the system will activate a... A transitional plan is proposed to ensure that service robots possess the necessary risk assessment capabilities even before a personalized baseline is established. The specific procedures of this transitional plan are as follows: When the system enters the active questioning mode and obtains a voice response during this initial phase, the operational logic of its dual-path parallel parsing mechanism will be adaptively adjusted. The first parsing path (linguistic content parsing) will continue to function normally; while the second parsing path (paralinguistic feature parsing) will temporarily adopt a universally applicable paralinguistic feature model pre-trained on a large-scale general dataset as a temporary baseline. To avoid misjudgments caused by individual differences in the universal model, the collaborative judgment rules at this stage will place greater emphasis on the analysis results of linguistic content. The judgment logic is as follows: If the linguistic content is clear and logical, it is prioritized as a safe state regardless of the deviation output of the universal model. If the linguistic content is unclear or meaningless (e.g., speech recognition failure or content chaos), the deviation of the universal model is then used as an auxiliary criterion. If the deviation exceeds the general threshold set for the universal model, it is collaboratively determined as a highly suspected medical emergency, and a corresponding alarm is triggered. This transitional scheme ensures that even without a personalized baseline, the system can make judgments based on a degraded but reliable logic that prioritizes linguistic content and uses secondary linguistic features in the initial stage of new user startup, effectively filling the security gap during the data accumulation period. In the initial stage, the system's computational model... In the standard monitoring mode, data is collected from all user-initiated, non-alarm-related daily voice interaction events, such as user voice commands to check the weather or control home appliances. For each collected voice signal, the system processor executes a signal processing algorithm to extract a set of quantified values ​​for paralinguistic features, including speech delay time, fundamental frequency perturbation, and pitch flatness. When the number of valid voice interaction samples collected reaches a preset statistical lower limit (set to 100), the system executes the initial baseline modeling procedure. The specific steps of the initial baseline modeling procedure are as follows: for each paralinguistic feature of the 100 collected samples, the system calculates its statistical mean. with standard deviation and this group The initial baseline of the user's personal normal values ​​is stored in the local database. As a specific numerical example, if the statistical mean of the 100 collected fundamental frequency perturbation samples is 0.3% and the standard deviation is 0.05%, then the baseline for this feature is determined as follows: Subsequently, the baseline model was officially adopted.

[0048] In subsequent operation, when the system experiences cognitive uncertainty... When the system exceeds the limit and enters the active questioning mode and obtains the user's voice response, the system first extracts the paralinguistic feature values ​​of the real-time response. Then, the deviation of this real-time value from the baseline mean is quantified by calculating the Z-score, and the calculation method is set to... The high deviation judgment rule used for the final decision is set as follows: when the absolute value of the Z-score of any key paralinguistic feature is greater than 3, it is judged as a high deviation. This judgment is based on the fact that a Z-score greater than 3 is a low-probability event in a normal distribution, indicating that the current vocal features deviate from the individual's normal state. To ensure the long-term effectiveness of the baseline and adapt to the user's slow physiological changes, the system also includes a baseline timeliness guarantee and reconstruction mechanism. This mechanism operates using a sliding time window. Every day, the system merges the new daily interactive voice sample data collected that day into the historical database in the background, while discarding the oldest data from more than 14 days ago, and recalculates the mean of all paralinguistic features based on this updated dataset. with standard deviation This serves as a baseline for the individual's normal value for the next day; this continuous online learning and updating process enables the baseline model to automatically track the user's long-term trends.

[0049] Example 5: This example describes the construction and calibration procedure of the semantic conflict matrix for rapid bypass triggering of high-confidence data conflicts, as well as the selection logic of the challenge action in the proactive challenge toolbox, to ensure the reproducibility of this method in different deployment environments. To make the definition of the conflict matrix adaptable to different sensor combinations and application scenarios, the system uses an offline calibration procedure to fill the matrix with data before deployment. The steps of the offline calibration procedure are as follows: First, in a controlled test environment, the tester wears a health monitoring wristband and is within the monitoring range of the service robot; then, the tester simulates a series of physically interacting events according to a preset script. The process of generating exclusionary behavioral state combinations involves simulating a simulated person in front of a vision sensor being forced into a supine sleeping state via external commands while wristband data indicates a high heart rate standing state. In each conflict scenario, high-confidence behavioral labels independently output by each sensor modality are recorded, and the corresponding element value of this label in the semantic conflict matrix is ​​set to the highest level of conflict value of 1.0. For other label combinations without logical conflict, such as (visual: sitting posture, audio: TV sound), their conflict value is set to 0. By systematically performing tests covering all preset key conflict scenarios, a semantic conflict matrix driven by data for a specific sensor configuration can be generated.

[0050] Furthermore, in the active questioning mode, the logic for selecting a specific questioning action from the questioning toolbox is set as a set of rules with priority and progression. This set of rules uses voice interaction as the default questioning action with the first priority. Only when the first-priority voice interaction does not receive any response signal that can be captured by the microphone within a preset response waiting time, the system control logic automatically executes the second-priority questioning action, that is, executes environmental stimuli aimed at attracting the target's attention. If the second-priority environmental stimulus also does not receive any response that can be captured by the visual sensor within a preset time, the system executes the third-priority questioning action, that is, changes the physical observation perspective of the sensor. This hierarchical selection logic constitutes an ordered questioning sequence from low interaction cost to high interaction cost.

[0051] Example 6: In a scenario where a user is resting in bed at night, their health monitoring wristband malfunctions and continuously outputs high confidence (0.95) information about a high heart rate in a standing posture. Meanwhile, the visual sensor on the service robot simultaneously outputs high confidence (0.98) information about the user being in bed in a sleeping posture. At this time, the system, in the normal monitoring mode, immediately detects this sharp conflict by querying a predefined semantic conflict matrix. It then bypasses the normal accumulation process of the cognitive uncertainty index and directly switches the operating mode from the normal monitoring mode to the active questioning mode. The system first executes the first priority voice questioning. After receiving no response, it automatically upgrades and executes the second priority environmental stimulus, namely, slowly increasing the brightness of the bedside night light. The visual sensor then captures the slight body movement of the user caused by the change in light. Based on this, the system judges that the user has the ability to react to external stimuli, thus determining the final behavior recognition result as a safe state and recording this sensor conflict event for subsequent maintenance. Subsequently, the system switches the operating mode back to the normal monitoring mode.

[0052] To further verify the method claimed in this invention and to demonstrate its beneficial effects on solving core technical problems compared to existing technologies, the following comparative examples are provided.

[0053] This comparative example uses a multimodal behavior recognition model based on post-fusion to compare its performance with the specific implementation method. This conventional technical model is identical to the specific implementation method in hardware configuration, both using a visual sensor and a millimeter-wave radar as data input sources. The core difference lies in the fact that the model in this comparative example does not possess a cognitive uncertainty index. Instead of a modal switching mechanism, it continuously runs in a single passive data fusion and analysis process. This process uses a weighted average algorithm known in the art to fuse the preliminary identification results of each independent modality to arrive at the final conclusion. To verify the processing capability of this conventional technical model in handling real data timing disorder scenarios caused by common reasons such as network latency and concurrent writing of multiple nodes, the experiment was conducted on an experimental platform simulating a home environment. This platform can synchronously input the exact same data stream to the experimental group processing unit using the method of this invention (its performance is shown in Example 2) and the processing unit using the comparative method of this invention. The experiment selected two test scenarios defined in Example 2: loss of consciousness in dangerous silence and unclear speech in dangerous silence. Each scenario was tested 10 times, and the judgment results and response time of the comparative method of this invention were recorded.

[0054] In a dangerous, silent loss-of-consciousness scenario, test subjects sat still for two minutes, simulating loss of consciousness. Their body posture remained largely unchanged, but they showed no response to external stimuli. The data stream received by the model in this comparative test was as follows: continuous output from the visual sensor (still, confidence level 0.75), and continuous output from the millimeter-wave radar sensor (stable breathing, confidence level 0.92). The model's weighted fusion algorithm, based on preset weights, continuously classified the final behavior as stillness, and the fused confidence level remained stable at 0.86. During the test period exceeding 600 seconds, because the input data stream showed no changes in event characteristics, the model failed to trigger any alarms in its system logs. In the experiment, only over 120 entries were recorded as user states without distinction: static affirmative records caused significant interference to subsequent manual error correction. In dangerous and silent scenarios with unclear speech, when the robot in the experimental group of this invention initiated a voice interrogation, although the model in this comparative example could also collect the user's ambiguous voice through the audio sensor, its processing flow was limited to attempting to perform standard speech recognition on the voice. Since it could not parse clear linguistic content from it, the model judged this audio event as unrecognizable environmental noise and ignored it. Its final judgment result was still static, which was derived from the fusion of visual and radar data, and it also failed to trigger any alarm.

[0055] Table 2: Performance comparison of Comparative Example 1 and the method of the present invention in key scenarios.

[0056] Test Scenario Computational model Final judgment result Alarm delay (seconds) Loss of consciousness Comparative Example 1 Method still >600 (Not triggered) Loss of consciousness Method of the present invention High-risk status 95.2 Speech unclear Comparative Example 1 Method still >600 (Not triggered) Speech unclear Method of the present invention Highly suspected medical emergency 108.6

[0057] Experimental results show that, in the absence of the cognitive uncertainty index When the mechanism drives the switching between conventional monitoring mode and active inquiry mode, the conventional passive data fusion model, due to the limitations of its design principle, is indeed unable to distinguish between a long-term featureless dangerous dormant state and a normal static state, thus there is a possibility of missed detection of key risks.

[0058] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A service robot behavior recognition method based on multimodal perception fusion, characterized in that, The method includes the following steps: Step a: The service robot is operated in a regular monitoring mode. In this mode, it continuously receives sensor data from at least two modes and sets a first threshold and a time window. Then, the calculation rule for the cognitive uncertainty index is executed. The calculation rule is as follows: when no mode can independently output a concern behavior label with a confidence level higher than the first threshold within the time window, and at least one mode continuously outputs concern behavior labels representing a neutral state, a cognitive uncertainty index increases cumulatively over time. Step b: Set a second threshold and establish a mode switching rule. The mode switching rule is: if and only if the index of cognitive uncertainty exceeds the second threshold, switch the operation mode of the method from the regular monitoring mode to the active questioning mode. Step c: In the active questioning mode, pause the fusion analysis of sensor data and control the service robot to perform non-command-based context-related voice interaction to obtain the user's response information to the voice interaction; Step d: Based on the user's response information, determine the final behavior recognition result, and switch the operation mode of the method back to the regular monitoring mode according to the mode switching rules; Furthermore, in step c, when the user's response is a voice response, step d, which involves determining the final behavior recognition result based on the response information, further includes: performing dual-path parallel parsing on the voice response, wherein the first parsing path is used to determine the linguistic content of the voice response, and the second parsing path is used to extract a set of paralinguistic features from the original audio waveform of the voice response; comparing the extracted set of paralinguistic features with the user's personal normal value baseline to obtain the deviation, the user's personal normal value baseline being established based on the user's voice data continuously collected and updated in daily interactions; and collaboratively determining the final behavior recognition result based on the linguistic content and the deviation. Before performing dual-path parallel parsing on the speech response, the method further includes: using a visual sensor to determine the three-dimensional spatial position of the user's mouth in the service robot's own coordinate system; based on the three-dimensional spatial position, controlling the microphone array to perform an acoustic beamforming algorithm to form an auditory focus pointing to the three-dimensional spatial position, so as to amplify the sound signal from the target direction and suppress environmental noise from other directions when acquiring the speech response, and using the audio signal generated after this processing as the speech response for dual-path parallel parsing; The method also includes: under the conventional monitoring modality, real-time detection of whether there are at least two modalities whose output behavioral information constitutes a conflict in the predefined semantic conflict matrix, and whose confidence scores are all higher than the conflict confidence threshold; if a conflict is detected, the calculation of the cognitive uncertainty index is bypassed, and the operation modality of the method is directly switched from the conventional monitoring modality to the active questioning modality.

2. The service robot behavior recognition method based on multimodal perception fusion according to claim 1, characterized in that, The method further includes: establishing a personalized circadian rhythm baseline for the user based on long-term accumulated, timestamped sensor data under conventional monitoring modalities, which depicts the user's typical behavioral patterns at different times of the day; in step b, the second threshold is dynamically adjusted according to the current time point and the personalized circadian rhythm baseline, and the adjustment rule is: the second threshold is increased during the time period when the user is usually in a static state according to the personalized circadian rhythm baseline, and decreased during the time period when the user is usually in an active state.

3. The service robot behavior recognition method based on multimodal perception fusion according to claim 1, characterized in that, The second parsing path extracts a set of paralinguistic features, including at least one of the following: the vocalization delay time from the end of the robot's question to the start of the user's response, the fundamental frequency perturbation characterizing the stability of vocal cord vibration, and the intonation flatness characterizing the richness of intonation variation.

4. The service robot behavior recognition method based on multimodal perception fusion according to claim 1, characterized in that, In step c, if no response information is obtained after performing non-command-based context-related voice interaction, the service robot is further controlled to perform at least one of the following actions: performing environmental stimuli aimed at attracting the target's attention, or changing the physical observation perspective of the sensors.

5. The service robot behavior recognition method based on multimodal perception fusion according to claim 1, characterized in that, In step d, when the response information indicates that there is no effective response to the proactive challenge, the final behavior identification result is determined to be a high-risk state, and an external alarm is triggered.

6. The service robot behavior recognition method based on multimodal perception fusion according to claim 2, characterized in that, In the step of dynamically adjusting the second threshold, the adjusted threshold Determined by the following rules: ,in, Based on the threshold, The modulation factor is determined based on the current time. With the determination of the personalized rhythm baseline, if the current time If it falls within the resting period defined by the personalized rhythm baseline, then If the current time is positive, If it falls within the active period defined by the personalized rhythm baseline, then It is a negative value.

7. The service robot behavior recognition method based on multimodal perception fusion according to claim 1, characterized in that, In the step of collaboratively determining the final behavior recognition result, the following collaborative judgment rules are established: when the linguistic content is clear and the deviation is below the deviation threshold, it is determined to be a safe state, and the modality switching rule is executed; when the linguistic content is clear but the deviation is above the deviation threshold, it is determined to be an abnormal state of voice features, and a specific alarm is triggered; when the linguistic content is unclear and the deviation is above the deviation threshold, it is determined to be a highly suspected medical emergency, and the highest level alarm is triggered.

Images