Intelligent man-machine collaborative interaction method and system based on multi-modal recognition

Abstract

The invention relates to the technical field of man-machine interaction, in particular to an intelligent man-machine cooperative interaction method and system based on multi-modal recognition. According to the method, multi-dimensional features of a user during voice interaction and gesture interaction are collected, wherein the multi-dimensional features comprise voice parameters, gesture parameters and situation parameters. When it is detected that the recognition confidence is lower than a threshold value, a state judgment process is triggered to judge that current interaction quality reduction belongs to basic capability limitation or temporary capability reduction. And when the basic capability is determined to be limited, constructing an extended path strategy cue word, inputting the extended path strategy cue word into a large model, and generating a multi-round progressive confirmation interaction instruction and an equipment control instruction by the large model. And when it is judged that the temporary capability declines, constructing a simplified path strategy cue word input large model, and generating a direct control instruction based on situational reasoning by the large model. And finally, interacting with the user according to the interaction instruction, and controlling the target equipment to execute operation according to the equipment control instruction. According to the method provided by the invention, the user experience and the interaction efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to an intelligent human-computer collaborative interaction method and system based on multimodal recognition. Background Technology

[0002] In smart home systems, voice commands combined with gestures have become the mainstream interaction method for users to control devices. This multimodal interaction method combines the convenience of voice with the intuitiveness of gestures, providing a more natural control experience. Existing technologies process user input through separate voice recognition and gesture recognition modules to obtain recognition results and confidence scores.

[0003] In practical use, users' speech clarity and gesture accuracy may decline for various reasons, leading to reduced recognition confidence or poor interaction quality. Different users exhibit varying interaction behaviors in different scenarios, requiring the system to adopt appropriate interaction strategies to address these situations.

[0004] Therefore, existing technologies still have some problems. They are not adaptable enough to the complex and ever-changing user states and usage scenarios, which can easily lead to poor user experience and low interaction efficiency. Summary of the Invention

[0005] To address, or at least partially address, the aforementioned technical problems, this application provides an intelligent interaction method based on multimodal recognition, which can improve user experience and interaction efficiency.

[0006] The first aspect is an intelligent human-computer collaborative interaction method based on multimodal recognition, which includes the following steps:

[0007] Collect multidimensional features when users perform voice and gesture interactions, including voice parameters, gesture parameters, and contextual parameters;

[0008] Obtain confidence scores for speech recognition and gesture recognition;

[0009] When the confidence level is detected to be lower than the threshold, the state determination process is triggered.

[0010] The state determination process determines whether the current decline in interaction quality is due to a limitation of basic capabilities or a temporary decline in capabilities.

[0011] When it is determined that the basic capability is limited, an extended path strategy prompt is constructed, and the extended path strategy prompt is input into the large model. The large model then generates multi-round progressive confirmation interactive instructions and device control instructions.

[0012] When a temporary capability decline is determined, a simplified path strategy prompt is constructed, and the simplified path strategy prompt is input into the large model, which then generates a direct control command based on contextual reasoning.

[0013] The system interacts with the user according to the interaction instructions and controls the target device to perform operations according to the device control instructions.

[0014] Optionally, the state determination process includes:

[0015] Input the multidimensional feature data of the current interaction into the evaluation-adjustment model;

[0016] The evaluation head analyzes the features of each dimension and generates structured evaluation text, which explains whether the voice parameters or gesture parameters reflect the true signal or are subject to temporary interference in the current scene.

[0017] By adjusting the head to receive the evaluation text and assigning dynamic weights to each dimension, disturbed dimensions receive low or negative weights, while reliable dimensions receive high weights.

[0018] Based on the dynamic weights and current features, the state type is determined, and the result of the judgment that the basic ability is limited or the temporary ability is reduced is output.

[0019] Optionally, determining the state type based on the dynamic weights and current features includes:

[0020] Obtain user historical baseline characteristics;

[0021] Input the current feature value, the historical baseline feature value, and the dynamic weight into the state discrimination network;

[0022] The deviation of the current feature value from the historical baseline feature value is calculated through the state discrimination network, and the deviation is adjusted in conjunction with the dynamic weights.

[0023] The state discrimination network outputs a state discrimination score based on the adjusted deviation.

[0024] The judgment result of basic ability limitation or temporary ability reduction is determined based on the state discrimination score.

[0025] Optionally, the state discrimination network includes:

[0026] The input layer receives the current feature vector, the historical baseline feature vector, and the dynamic weight vector.

[0027] The deviation calculation layer, connected to the input layer, is used to receive the current feature value vector and the historical baseline feature value vector, calculate the normalized deviation of each dimension through element-level subtraction and normalization operations, and output the deviation vector.

[0028] A weight adjustment layer, connected to the deviation calculation layer and the input layer, is used to receive the deviation vector and the dynamic weight vector, and fuse the deviation vector and the dynamic weight vector through an element-wise multiplication gate mechanism to output a weighted deviation vector.

[0029] The dimension aggregation layer, connected to the weight adjustment layer, is used to receive the weighted deviation vector and divide it into core interaction dimension group and context dimension group according to dimension type. The core interaction aggregation feature vector and context aggregation feature vector are extracted through independent fully connected sub-networks, and the concatenated vector of the two aggregation features is output.

[0030] The state discrimination layer, connected to the dimension aggregation layer, receives the concatenated vector, processes the concatenated vector through a multi-layer fully connected network, and outputs a state discrimination score.

[0031] Optionally, in the weight adjustment layer, when the dynamic weight is positive, the influence of the corresponding dimension deviation is amplified; when the dynamic weight is negative, the influence of the corresponding dimension deviation is reversed or suppressed.

[0032] Optionally, the core interaction dimension group of the dimension aggregation layer includes voice parameters and gesture parameters, and the context dimension group includes environmental noise level, user location, dwell time at the location, and lighting conditions.

[0033] Optionally, the state discrimination layer learns the mapping relationship between state types and deviation patterns. When the core interaction aggregation feature shows low deviation and the context aggregation feature shows low anomaly, the output score approaches 0, and when the context aggregation feature shows high anomaly, the output score approaches 1.

[0034] Secondly, this application proposes an intelligent human-computer collaborative interaction system based on multimodal recognition, the system comprising at least one module, the at least one module being used to execute an intelligent human-computer collaborative interaction method based on multimodal recognition as described in any of the first aspects.

[0035] The technical solution provided in this application has the following advantages compared with the prior art:

[0036] One of its beneficial effects and its working principle is as follows:

[0037] Existing multimodal interaction systems employ a uniform fault-tolerance strategy when dealing with low recognition confidence, requiring users to re-enter information for more precise expressions. For users with limited basic abilities (those who consistently exhibit insufficient operational precision in everyday, low-interference scenarios due to physiological conditions or individual differences), re-entry still results in the same limitation, leading to recognition failure. For users with temporary cognitive decline (those whose cognitive resources are occupied by the primary task when performing tasks requiring high attention or handling emergencies, causing a temporary decrease in the precision of interactive operations), requiring more precise input further increases the cognitive burden. The uniform strategy fails to effectively address these two different sources of decline.

[0038] This application collects multi-dimensional features from user voice and gesture interactions and uses a state discrimination process to determine whether the current interaction quality degradation is due to a limitation in basic capabilities or a temporary degradation. When it is determined to be a limitation in basic capabilities, a large-scale model for extended path strategy prompts is constructed. Each round, a simple binary confirmation question is generated along with visual feedback. Based on the user's feedback in each round, the candidate range is gradually narrowed down, ultimately achieving precise device control commands. When it is determined to be a temporary degradation, a large-scale model for simplified path strategy prompts is constructed. This large-scale model integrates contextual information for reasoning, skips the confirmation step, directly generates control commands, and provides brief voice feedback.

[0039] Therefore, for users with limited basic skills, by extending the interaction path and using multiple rounds of progressive confirmation, what users cannot complete in a single precise operation is transformed into a cumulative process of multiple simple confirmations, preventing users from repeatedly failing due to insufficient operational precision. For users with temporary decline in skills, by simplifying the interaction path and contextual reasoning, the cognitive resources required by users are reduced, eliminating the need for them to stop their main tasks to handle complex interaction processes. Differentiating the sources of decline allows the system to apply matching strategies, using opposite strategies for different sources, avoiding the deterioration of performance caused by strategy mismatch.

[0040] Its second beneficial effect and its working principle are as follows:

[0041] Determining the type of user interaction state requires comprehensive multi-dimensional feature information. When certain feature dimensions are affected by temporary environmental factors, if each feature dimension participates in the determination with a fixed weight, the feature anomalies caused by the interference factor may be misjudged as signals of the user's true state. For example, reduced speech clarity in a noisy environment might be interpreted as evidence of limited basic capabilities using a fixed-weight approach, when in reality it is simply a problem with the acquisition quality caused by environmental interference. The lack of dynamic adjustment of the weights for each dimension makes it easy to be misled by temporary interference factors, reducing the accuracy of state determination.

[0042] This application provides a multi-dimensional feature dynamic weighting discrimination method based on evaluation and adjustment. The evaluation head receives complete feature data of the current interaction and generates structured interaction state evaluation text. The evaluation text describes the feature values, the influence of contextual factors, the possible causes of the feature in that dimension, whether it reflects the true signal or is subject to temporary interference, and analyzes the support or contradiction relationships between dimensions. The adjustment head receives the evaluation text generated by the evaluation head and assigns dynamic importance weights to each dimension based on the reliability analysis in the evaluation text. Dimensions subject to interference receive low or negative weights, while reliable dimensions receive high weights. The weights are dynamically calculated based on the current scenario and the analysis results of the evaluation text. The negative weighting mechanism ensures that dimensions providing reverse evidence in specific scenarios can correctly participate in subsequent calculations.

[0043] Therefore, the weights of each feature dimension are dynamically adjusted according to the specific scenario, with the weights of disturbed dimensions decreasing and the weights of reliable dimensions increasing, thus avoiding being misled by temporary environmental interference factors. This distinguishes between the user's true state signal and environmental interference signal, improving the system's robustness in harsh environments and providing accurate weighting criteria for subsequent state determination.

[0044] Its third beneficial effect and its working principle are as follows:

[0045] Users with limited basic capabilities have consistently underperformed in normal scenarios, but they may also experience temporary declines in their abilities, such as an elderly user suddenly needing to control equipment while cooking in the kitchen.

[0046] If we only consider historical baselines to determine whether a user's basic ability is limited or their ability is temporarily declining, when a decline in the interaction quality of a user with limited basic abilities is detected, the system will easily categorize them as having limited basic abilities because their historical baseline is already poor. The dominant factor is the inability to recognize anomalies in the current context. This leads the system to still employ an extended path strategy requiring multiple rounds of confirmation when a user with limited basic abilities encounters a scenario of temporary ability decline. This further increases the cognitive burden on the user in an emergency situation where their cognitive resources are already occupied, ignoring the objective fact that users with limited basic abilities also face temporary scenarios that require rapid responses.

[0047] This application provides a deviation-aware state discrimination network. A deviation calculation layer explicitly calculates the deviation of each dimension's current value relative to the historical baseline, allowing historical baseline information to directly participate in the discrimination. Normalization processing ensures comparability between dimensions of different dimensions. A weight adjustment layer uses a multiplication gate mechanism to dynamically adjust the deviation; positive dynamic weights amplify the deviation, while negative dynamic weights suppress it, solving the problem of erroneously amplifying the deviation of disturbed dimensions. The grouped fully connected structure of the dimension aggregation layer processes the core interaction dimension group and the context dimension group separately, capturing specific patterns of different dimension groups. The sub-network of the core interaction dimension group learns to extract collective consistency patterns, while the sub-network of the context dimension group learns to extract mutation patterns. The state discrimination layer performs deep inference based on the two sets of aggregated features. When the core interaction aggregated features show low deviation and the context aggregated features show low anomaly, it is determined that the basic ability is limited; when the context aggregated features show high anomaly, it is determined that the temporary ability is reduced. In this case, even if the core interaction aggregated features show high deviation, it is interpreted as a temporary state influence. This discrimination logic addresses the issue of users with limited basic capabilities encountering temporary states. When the weighted deviation of the context dimension is high, the state discrimination layer will determine that the temporary capability has decreased, thus avoiding the error of fixedly judging the user's basic capability as limited simply because the user's baseline capability is limited.

[0048] Therefore, it solves the problem of misjudgment when users with limited basic capabilities encounter a temporary decline in capabilities, taking into account the actual situation that users with limited basic capabilities also need to respond quickly. Attached Figure Description

[0049] Figure 1 A schematic diagram illustrating an application scenario of the intelligent human-computer collaborative interaction method based on multimodal recognition provided in this application embodiment;

[0050] Figure 2 One of the flowcharts of the intelligent human-computer collaborative interaction method based on multimodal recognition provided in the embodiments of this application;

[0051] Figure 3 The second flowchart illustrates the intelligent human-computer collaborative interaction method based on multimodal recognition provided in this application embodiment.

[0052] Figure 4 The third flowchart illustrates the intelligent human-computer collaborative interaction method based on multimodal recognition provided in this application embodiment;

[0053] Figure 5 The fourth flowchart illustrates the intelligent human-machine collaborative interaction method based on multimodal recognition provided in this application embodiment. Detailed Implementation

[0054] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0056] Before providing a detailed explanation of the embodiments of this application, let's first introduce the application scenarios involved in the embodiments of this application.

[0057] like Figure 1 As shown in the application scenario provided in this application embodiment, after the smart home system receives the user's voice and gesture interaction commands, the intelligent human-machine collaborative interaction method based on multimodal recognition can be applied to scenarios where the interaction quality deteriorates. For example, when a user controls a device by issuing voice commands in conjunction with gesture operations, if a low recognition confidence level or a decline in interaction quality is detected, the intelligent human-machine collaborative interaction device based on multimodal recognition determines whether the current decline in interaction quality is due to a limitation of basic capabilities or a temporary decline in capabilities. Based on the determination result, a differentiated interaction strategy is adopted to generate device control commands, ultimately achieving precise control of the target device. Specifically, the intelligent human-machine collaborative interaction method based on multimodal recognition provided in this application embodiment can be executed by an intelligent human-machine collaborative interaction system based on multimodal recognition. The system includes at least one module, which is used to execute the intelligent human-machine collaborative interaction method based on multimodal recognition described in the following embodiments.

[0058] Reference Figure 2 As shown, the intelligent human-computer collaborative interaction method based on multimodal recognition described in this application includes the following steps:

[0059] S201: Collect multi-dimensional features when the user performs voice and gesture interactions, the multi-dimensional features including voice parameters, gesture parameters and contextual parameters;

[0060] Specifically, the system collects the user's voice signal through a microphone array, extracting voice parameters including volume, speech rate, pause duration, and pronunciation clarity. It also collects the user's hand gestures through a camera or depth sensor, extracting gesture parameters including gesture jitter amplitude, gesture pointing duration, and gesture pointing offset distance. Simultaneously, it collects contextual parameters, including ambient noise level, user location, user dwell time at that location, and lighting conditions.

[0061] S202: Obtain the confidence scores for speech recognition and gesture recognition;

[0062] Specifically, the collected speech signal is input into the speech recognition module for speech recognition. The speech recognition module calculates the probability distribution of the recognition result based on the speech recognition model (such as SenseVoice) and outputs the most likely recognized text and the corresponding speech recognition confidence value.

[0063] Simultaneously, the collected gesture data is input into the gesture recognition module for gesture recognition. The gesture recognition module calculates the probability distribution of gesture types based on a gesture feature recognition model (such as YOLO) and outputs the recognized gesture type and the corresponding gesture recognition confidence value.

[0064] The confidence score reflects the reliability of the recognition results. A low confidence score indicates that the speech clarity or gesture accuracy of the current interaction is insufficient.

[0065] S203: When any of the confidence values ​​is detected to be lower than the threshold, the state determination process is triggered;

[0066] Specifically, preset confidence thresholds for speech recognition and gesture recognition are established. When the confidence level of speech recognition is lower than the preset confidence threshold, or the confidence level of gesture recognition is lower than the preset confidence threshold, it is determined that the current interaction quality has declined, triggering the state judgment process.

[0067] Record all multidimensional feature data collected at the trigger moment, including voice parameters, gesture parameters, and context parameters. Use this data as input to the state judgment process to determine whether the current decline in interaction quality is due to a limitation of basic capabilities or a temporary decline in capabilities.

[0068] S204: The state determination process determines that the current decline in interaction quality is due to either a limitation of basic capabilities or a temporary decline in capabilities.

[0069] Reference Figure 3 As shown, in this embodiment of the application, S204 includes the following steps:

[0070] S301: Input the multidimensional feature data of the current interaction into the evaluation-adjustment model;

[0071] Specifically, the multi-dimensional feature data collected at the trigger moment is organized into an input vector. Voice parameters include volume, speech rate, pause duration, and pronunciation clarity score; gesture parameters include tremor amplitude, pointing duration, and pointing offset distance; contextual parameters include ambient noise level, user location identifier, dwell time, and lighting conditions. These parameters are then concatenated according to a predefined feature order to form a feature vector.

[0072]

[0073] in Represents a subvector of speech parameters. Represents a subvector of gesture parameters. This represents a subvector of contextual parameters. This feature vector is then input into an evaluation-adjustment model based on a large language model. This model includes two processing modules: an evaluation head and an adjustment head. The feature vector first enters the evaluation head for reliability evaluation.

[0074] S302: Analyze the features of each dimension through the evaluation head and generate structured evaluation text, wherein the evaluation text describes whether the voice parameters or gesture parameters reflect the real signal or are subject to temporary interference in the current scene;

[0075] Specifically, the evaluation head receives the feature vector. Next, a large language model is constructed to analyze the input of the prompt words. These prompt words require the model to analyze the reasons for the generation of features along different dimensions.

[0076] Regarding the characteristic of low speech intelligibility, the evaluation head can analyze its cause through the following prompts: Based on the user's location and dwell time, if the current ambient noise level is within the normal range, then low speech intelligibility reflects the user's limited pronunciation ability and is judged as a real signal; if the current ambient noise level is abnormal, then low speech intelligibility is mainly caused by ambient noise interference and is judged as temporary interference.

[0077] Regarding the characteristic of large gesture jitter, the evaluation head can analyze its cause through the following prompts: Based on the user's location and dwell time, if the current lighting conditions are stable and sufficient, the gesture jitter reflects the user's limited gesture control ability and is judged as a real signal; if the current lighting conditions change drastically or are insufficient, the gesture jitter is mainly caused by the instability of the recognition system and is judged as temporary interference.

[0078] Based on these causal analyses, the large language model generates structured evaluation text, which describes the feature values, the influence of contextual factors, the possible causes of the feature in that dimension, whether it reflects the true signal or is subject to temporary interference, and the support or contradiction relationships between the dimensions, outputting a complete reliability evaluation result.

[0079] S303: The evaluation text is received by adjusting the head and dynamic weights are assigned to each dimension. Disturbed dimensions receive low or negative weights, while reliable dimensions receive high weights.

[0080] Specifically, after the adjustment head receives the structured evaluation text generated by the evaluation head, a large language model with weighted input prompt words is constructed.

[0081] The prompt requires the model to assign numerical, dynamic weights to each dimension based on the reliability analysis in the evaluation text.

[0082] For example, a prompt word example is:

[0083] Dynamic weights are assigned to the speech and gesture dimensions based on the following evaluation text, with weight values ​​ranging from (-1, 1):

[0084] For dimensions deemed to reflect the user's true state, positive weights are assigned, with the weight reflecting the degree of reliability.

[0085] For dimensions identified as being subject to temporary interference, negative weights are assigned, with the weight size reflecting the degree of interference.

[0086] Evaluation text:

[0087] [Insert structured evaluation text generated by the evaluation header here]

[0088] Please output the weights for the voice dimension and the gesture dimension.

[0089] The language model outputs dynamic weight vectors for each dimension.

[0090] S304: Based on the dynamic weights and current features, determine the state type and output the discrimination result of limited basic capabilities or temporary decreased capabilities.

[0091] See Figure 4 As shown, in this embodiment of the application, S304 includes the following steps:

[0092] S401: Obtain user's historical baseline characteristics;

[0093] Specifically, historical baseline features are extracted from the user's historical interaction database within a preset historical data time window. Historical baseline features refer to the statistical average of various dimensions of a user's characteristics under normal interaction scenarios, used to characterize the user's level of normal interaction ability.

[0094] A preset historical data time window is used; in this embodiment, interaction records from the most recent 30 days are extracted. Data from normal interaction scenarios is filtered out, excluding interaction records with abnormal contextual dimensions.

[0095] Specific screening criteria include: ambient noise level is below a preset noise threshold, such as 60 decibels; lighting conditions are within a normal range, such as 100 to 500 Lux; and the user stays at the location for more than a preset duration, such as 2 minutes.

[0096] S402: Input the current feature value, the historical baseline feature value, and the dynamic weight into the state discrimination network.

[0097] S403: The deviation of the current feature value from the historical baseline feature value is calculated through the state discrimination network, and the deviation is adjusted in combination with the dynamic weights.

[0098] S404: The state discrimination network outputs a state discrimination score based on the adjusted deviation.

[0099] Specifically, state discrimination networks include:

[0100] S501: Input layer, used to receive the current feature vector, historical baseline feature vector, and dynamic weight vector.

[0101] S502: Deviation calculation layer, connected to the input layer, is used to receive the current feature value vector and the historical baseline feature value vector, calculate the normalized deviation of each dimension through element-level subtraction and normalization operations, and output the deviation vector.

[0102] Specifically, the deviation calculation layer receives the current feature value vector and the historical baseline feature value vector from the input layer. It first performs a subtraction operation on corresponding elements of the two vectors to calculate the original deviation of the current feature value relative to the historical baseline feature value in each dimension. The original deviation reflects the difference between the user's current performance and their historical normal level. The deviation calculation layer normalizes the standard deviation of the original deviation in each dimension, making dimensions with different dimensions comparable. The deviation calculation layer organizes the normalized deviations of each dimension of speech parameters, gesture parameters, and contextual parameters into a complete deviation vector. This vector contains deviation information for all dimensions and is passed as the output of the deviation calculation layer to the weight adjustment layer. By explicitly calculating the deviation, historical baseline information can directly participate in state discrimination, which is a key basis for distinguishing between basic ability limitations and temporary ability decline.

[0103] The deviation calculation layer does not contain trainable parameters.

[0104] S503: Weight adjustment layer, connected to the deviation calculation layer and the input layer, for receiving the deviation vector and the dynamic weight vector, fusing the deviation vector and the dynamic weight vector through an element-wise multiplication gate mechanism, and outputting a weighted deviation vector.

[0105] Specifically, the weight adjustment layer receives the deviation vector output by the deviation calculation layer and the dynamic weight vector passed in from the input layer. The dynamic weight vector only contains the speech dimension weight and gesture dimension weight, excluding the context dimension weight. The weight adjustment layer adjusts the deviation of the core interaction dimensions through an element-wise multiplication gate mechanism. It multiplies the normalized deviation of the speech dimension with the dynamic weight of the speech dimension to obtain the weighted deviation of the speech dimension, and multiplies the normalized deviation of the gesture dimension with the dynamic weight of the gesture dimension to obtain the weighted deviation of the gesture dimension. When the dynamic weight is positive, the deviation of that dimension is amplified, enhancing its influence on state discrimination; when the dynamic weight is negative, the deviation of that dimension is reversed or suppressed, reducing or reversing its influence. For the context dimension, the weight adjustment layer does not perform weight adjustment and directly retains its original normalized deviation. The weight adjustment layer organizes the adjusted weighted deviations of the speech and gesture dimensions, along with the unadjusted context dimension deviation, into a weighted deviation vector and passes it to the dimension aggregation layer. The dynamic weight adjustment mechanism avoids misjudging deviations caused by environmental interference as changes in user status.

[0106] The weight adjustment layer does not contain trainable parameters.

[0107] S504: Dimension aggregation layer, connected to the weight adjustment layer, is used to receive the weighted deviation vector and divide it into core interaction dimension group and context dimension group according to dimension type. It extracts core interaction aggregation feature vector and context aggregation feature vector through independent fully connected sub-networks, and outputs the concatenated vector of the two aggregation features.

[0108] Specifically, the dimension aggregation layer receives the weighted deviation vector output by the weight adjustment layer and divides it into two groups according to the dimension type for independent processing.

[0109] The dimensional aggregation layer extracts the speech and gesture parameters from the weighted deviation vector to form the core interaction dimension sub-vector. Speech parameters include weighted deviations for volume, speech rate, pauses, and pronunciation clarity; gesture parameters include weighted deviations for jitter amplitude, pointing duration, and pointing offset. The dimensional aggregation layer inputs the core interaction dimension sub-vector into the fully connected subnetwork of the core interaction dimension group. This subnetwork performs a linear transformation on the input using a trainable weight matrix and bias vector, followed by activation function processing. It learns to extract the collective consistency pattern of the speech and gesture dimensions, i.e., whether speech and gesture simultaneously exhibit a sustained low level, and outputs the core interaction aggregation feature vector.

[0110] The dimensional aggregation layer extracts the contextual parameter portion from the weighted deviation vector to form a contextual dimension sub-vector. Contextual parameters include the deviation from environmental noise level, user location, dwell time, and lighting conditions. The dimensional aggregation layer inputs the contextual dimension sub-vector into a fully connected subnetwork of the contextual dimension group. This subnetwork performs a linear transformation on the input using independent weight matrices and bias vectors, followed by activation function processing. It learns to extract the degree of deviation between the contextual dimensions and the user's usual scenarios, i.e., it judges the degree of inconsistency between the current contextual parameters and the user's historical normal scenarios, and outputs a contextual aggregated feature vector.

[0111] The dimensional aggregation layer concatenates the core interaction aggregation feature vector and the context aggregation feature vector to form a concatenated vector, which is then passed to the state discrimination layer. Through a grouped fully connected structure, specific patterns of the two sets of dimensions are extracted independently, effectively capturing the collective consistency information of the core interaction dimension and the deviation information of the context dimension from typical scenarios. This provides the state discrimination layer with key features to distinguish between the two states.

[0112] S505: State discrimination layer, connected to the dimension aggregation layer, for receiving the concatenated vector, processing the concatenated vector through a multi-layer fully connected network and outputting a state discrimination score.

[0113] Specifically, the state discrimination layer receives the concatenated vector output by the dimension aggregation layer. This concatenated vector contains both core interaction aggregation features and context aggregation features. The state discrimination layer performs deep feature extraction on the concatenated vector through a multi-layer fully connected network. Each fully connected network layer undergoes linear transformation using a weight matrix and bias vector, followed by activation function processing to extract higher-level abstract feature representations layer by layer. The state discrimination layer learns the mapping relationship between state types and deviation patterns. Through training, it learns that when the core interaction aggregation features show low deviation and the context aggregation features show low deviation from the normal scenario, it determines that the basic ability is limited; when the context aggregation features show high deviation from the normal scenario, it determines that the temporary ability is reduced. In the final layer, the state discrimination layer outputs a state discrimination score through a sigmoid activation function. This score ranges from 0 to 1, with values ​​close to 0 indicating limited basic ability and values ​​close to 1 indicating a temporary reduction in ability. The key discrimination logic is that when the context aggregation feature shows a high deviation, even if the core interaction aggregation feature also shows a deviation, the state discrimination layer will output a score close to 1 to interpret it as a temporary state influence, thus solving the problem of misjudgment when users with limited basic capabilities encounter a temporary decline in ability.

[0114] During the training phase, a large amount of user interaction data is collected to form a training set. Each training sample contains the current feature value, historical baseline feature value, dynamic weight, and corresponding state type label.

[0115] The status type labels include two categories: basic ability limitation and temporary ability decline. The basic ability limitation samples are from user interaction records where the user consistently performs poorly in a stable environment, while the temporary ability decline samples are from user interaction records where the user's performance declines in a specific context.

[0116] The training set also needs to collect samples of users with limited basic abilities encountering temporary ability decline scenarios and label them as temporary ability declines. This is used to train the state discrimination network to learn the discrimination logic that prioritizes the judgment of temporary ability declines when the context dimension shows a high deviation.

[0117] By updating the trainable parameters of the dimension aggregation layer and the state discrimination layer through backpropagation of the training data, the network can learn the mapping relationship between state types and deviation patterns.

[0118] S405: Determine the judgment result of basic ability limitation or temporary ability reduction based on the state discrimination score.

[0119] Specifically, a preset state discrimination threshold is set, which is usually set to 0.5.

[0120] When the state discrimination score is less than the preset state discrimination threshold, the current interaction quality decline is determined to be due to a limitation in basic capabilities; when the state discrimination score is greater than the preset state discrimination threshold, the current interaction quality decline is determined to be due to a temporary decline in capabilities. The final state category discrimination result is output for subsequent interaction strategy selection.

[0121] S205: When it is determined that the basic capability is limited, construct an extended path strategy prompt, input the extended path strategy prompt into the large model, and the large model generates multi-round progressive confirmation interactive instructions and device control instructions;

[0122] Specifically, based on the judgment results of limited basic capabilities, strategy prompts for extending the path are constructed.

[0123] In this application embodiment, an example of the extended path strategy prompt is:

[0124] Strategy type: Multi-round progressive confirmation mode

[0125] Core principle: Break down the need for precise, one-time control into multiple simple binary confirmation operations, where each round only requires the user to answer "yes" or "no";

[0126] Current recognition results:

[0127] Speech recognition text: Turn off...lights (Confidence: 0.42);

[0128] Gesture pointing area: living room direction (confidence level: 0.38);

[0129] Candidate device list: [This data is obtained through a pre-configured home database. For example, candidate furniture devices for the living room include: living room ceiling light, living room table lamp, living room floor lamp, and dining room chandelier];

[0130] Please transform the above fuzzy input into a set of candidate devices, design a progressive confirmation path, and generate the first round of binary confirmation questions and visual feedback instructions.

[0131] The first round of interaction is executed, and user feedback is received. This feedback is then input into a large model to narrow down the candidate pool. Based on the user feedback, the large model generates a second round of confirmation questions for more granular filtering, such as whether the device is a lighting fixture, and generates new visual feedback instructions accordingly. Multiple rounds of progressive confirmation are continuously executed, with the large model gradually eliminating incorrect options based on accumulated confirmation information in each round, until the candidate device is narrowed down to a single target.

[0132] S206: When it is determined that there is a temporary decrease in capability, a simplified path strategy prompt is constructed, the simplified path strategy prompt is input into the large model, and the large model generates a direct control command based on contextual reasoning.

[0133] Specifically, simplified path strategy prompts are constructed based on the results of the temporary capability reduction assessment.

[0134] In this application embodiment, an example of the simplified path strategy prompt is:

[0135] Strategy type: Fast inference direct execution mode

[0136] Core principle: Skip the confirmation step; the system directly infers and outputs the most likely target device and operation, reducing user operation steps and decision-making frequency.

[0137] Current recognition results:

[0138] Speech recognition text: Close...music (confidence: 0.40);

[0139] Gesture pointing area: living room direction (confidence level: 0.35);

[0140] Context parameters:

[0141] User location: Kitchen;

[0142] Duration of stay: 5 minutes;

[0143] Ambient noise: 72 decibels (relatively high);

[0144] Lighting conditions: Normal;

[0145] Based on contextual information, infer the user's intent, directly output the target device and operation type, and generate control commands and brief execution feedback voice prompts.

[0146] S207: Interact with the user according to the interaction instructions, and control the target device to perform operations according to the device control instructions.

[0147] Specifically, the corresponding interaction process is executed based on the type of interaction command generated by the large model. When using an extended path strategy, the interaction with the user follows a multi-round progressive confirmation mode. First, based on the first round of interaction commands generated by the large model, the voice output module plays a question, such as, "Do you want to control the devices in your living room?" User feedback is received and passed to the large model for the next round of reasoning. This process continues with multiple rounds of interaction, playing the question and updating the visual feedback in each round based on the interaction commands generated by the large model, until the large model identifies a unique target device and generates the final device control command.

[0148] When a simplified path strategy is adopted, multiple confirmation rounds are skipped. After the large model directly generates the control command, a brief execution feedback is played through the voice output module, such as: "The living room music has been turned off for you," informing the user of the upcoming operation. The device control command is then sent to the target device via the smart home communication protocol, and the target device executes the corresponding operation upon receiving the command.

[0149] The device status feedback interface confirms the device execution result. If the device execution is successful, the interaction process ends.

[0150] For the simplified path strategy, a fast error correction activation state is initiated after the control command is sent, continuously listening for the user's error correction commands, with a listening time window set to 3 to 5 seconds. If an error correction command, such as incorrect or wrong, is detected from the user within the time window, an undo command is immediately sent to the target device via the smart home communication protocol to restore the device to its state before execution. At the same time, the error correction information is fed back to the large model to re-perform contextual reasoning and generate new control commands.

[0151] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0152] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application, "first," "second," and various numerical designations are only for ease of description and are not used to limit the scope of the embodiments of this application. For example, they are used to distinguish different messages, rather than to describe a specific order or sequence.

[0153] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0154] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for intelligent human-computer collaborative interaction based on multimodal recognition, characterized in that, Includes the following steps: Collect multidimensional features when users perform voice and gesture interactions, including voice parameters, gesture parameters, and contextual parameters; Obtain confidence scores for speech recognition and gesture recognition; When the confidence level is detected to be lower than the threshold, the state determination process is triggered. The state determination process determines whether the current decline in interaction quality is due to a limitation of basic capabilities or a temporary decline in capabilities. When it is determined that the basic capability is limited, an extended path strategy prompt is constructed, and the extended path strategy prompt is input into the large model. The large model then generates multi-round progressive confirmation interactive instructions and device control instructions. When a temporary capability decline is determined, a simplified path strategy prompt is constructed, and the simplified path strategy prompt is input into the large model, which then generates a direct control command based on contextual reasoning. The system interacts with the user according to the interaction instructions and controls the target device to perform operations according to the device control instructions.

2. The intelligent human-computer collaborative interaction method based on multimodal recognition according to claim 1, characterized in that, The state determination process includes: Input the multidimensional feature data of the current interaction into the evaluation-adjustment model; The evaluation head analyzes the features of each dimension and generates structured evaluation text, which explains whether the voice parameters or gesture parameters reflect the true signal or are subject to temporary interference in the current scene. By adjusting the head to receive the evaluation text and assigning dynamic weights to each dimension, disturbed dimensions receive low or negative weights, while reliable dimensions receive high weights. Based on the dynamic weights and current features, the state type is determined, and the result of the judgment that the basic ability is limited or the temporary ability is reduced is output.

3. The intelligent human-computer collaborative interaction method based on multimodal recognition according to claim 2, characterized in that, The process of determining the state type based on the dynamic weights and current features includes: Obtain user historical baseline characteristics; Input the current feature value, the historical baseline feature value, and the dynamic weight into the state discrimination network; The deviation of the current feature value from the historical baseline feature value is calculated through the state discrimination network, and the deviation is adjusted in conjunction with the dynamic weights. The state discrimination network outputs a state discrimination score based on the adjusted deviation. The judgment result of basic ability limitation or temporary ability reduction is determined based on the state discrimination score.

4. The intelligent human-computer collaborative interaction method based on multimodal recognition according to claim 3, characterized in that, The state discrimination network includes: The input layer receives the current feature vector, the historical baseline feature vector, and the dynamic weight vector. The deviation calculation layer, connected to the input layer, is used to receive the current feature value vector and the historical baseline feature value vector, calculate the normalized deviation of each dimension through element-level subtraction and normalization operations, and output the deviation vector. A weight adjustment layer, connected to the deviation calculation layer and the input layer, is used to receive the deviation vector and the dynamic weight vector, and fuse the deviation vector and the dynamic weight vector through an element-wise multiplication gate mechanism to output a weighted deviation vector. The dimension aggregation layer, connected to the weight adjustment layer, is used to receive the weighted deviation vector and divide it into core interaction dimension group and context dimension group according to dimension type. The core interaction aggregation feature vector and context aggregation feature vector are extracted through independent fully connected sub-networks, and the concatenated vector of the two aggregation features is output. The state discrimination layer, connected to the dimension aggregation layer, receives the concatenated vector, processes the concatenated vector through a multi-layer fully connected network, and outputs a state discrimination score.

5. The intelligent human-computer collaborative interaction method based on multimodal recognition according to claim 4, characterized in that, In the weight adjustment layer, when the dynamic weight is positive, the influence of the corresponding dimension deviation is amplified; when the dynamic weight is negative, the influence of the corresponding dimension deviation is reversed or suppressed.

6. The intelligent human-computer collaborative interaction method based on multimodal recognition according to claim 4, characterized in that, The core interactive dimension group of the dimension aggregation layer includes voice parameters and gesture parameters, and the context dimension group includes environmental noise level, user location, time spent at the location, and lighting conditions.

7. The intelligent human-computer collaborative interaction method based on multimodal recognition according to claim 4, characterized in that, The state discrimination layer learns the mapping relationship between state type and deviation pattern. When the core interaction aggregation feature shows low deviation and the context aggregation feature shows low anomaly, the output score approaches 0. When the context aggregation feature shows high anomaly, the output score approaches 1.

8. An intelligent human-computer collaborative interaction system based on multimodal recognition, characterized in that, The system includes at least one module, which is used to execute the intelligent human-machine collaborative interaction method based on multimodal recognition as described in any one of claims 1-7.

Images