An interactive intention recognition and risk prediction method fusing multi-modal perception

Abstract

The application discloses a kind of interactive intention recognition and risk prediction methods of fusion multimodal perception, it is related to human-computer interaction technical field, including: extracting the time sequence characteristics of operator multimodal perception signal, generates non-negative evidence vector by processing through the evidence coding network corresponding to each mode, constructs Dirichlet distribution simultaneously obtains intention category probability and cognitive uncertainty;Multi-modal evidence is fused in evidence space, generates uncertainty state containing joint intention distribution, joint cognitive uncertainty and conflict degree;Uncertainty state is combined with scene context and input risk prediction model, and the risk level that is up-regulated with the increase of uncertainty is output;When uncertainty overrun, intention confirmation is initiated, and enhanced evidence vector is re-fused to converge uncertainty.The application can improve intention recognition accuracy and risk decision security in human-machine shared control scene.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a method for interaction intent recognition and risk prediction that integrates multimodal perception. Background Technology

[0002] In human-machine shared control scenarios, existing interactive intent recognition and risk prediction generally adopt a serial processing architecture. First, intent recognition is completed based on the collected multimodal perception signals, and then a deterministic intent category label is output. The label is then input into the downstream risk prediction module to carry out risk assessment.

[0003] Under this architecture, the cognitive uncertainty information contained in the intent recognition process is completely discarded in the transmission stage. The risk prediction module cannot perceive the reliability of the upstream intent judgment and can only perform risk calculation based on the potentially biased intent result, resulting in information attenuation of the risk signal at the source and forming a cascading fault. At the same time, when there are discrepancies in multimodal perception signals, fluctuations in the quality of single-modal signals, or when the operator's intent is in a fuzzy state, the existing solution lacks an active uncertainty convergence mechanism, cannot quickly resolve excessive cognitive uncertainty, and cannot guarantee the stability and interactive security of intent judgment and risk decision-making. Summary of the Invention

[0004] This application provides a method for interaction intent recognition and risk prediction that integrates multimodal perception. It can solve the problems in related technologies, such as the inability of uncertainty in intent recognition under serial processing architecture to be transmitted to the risk prediction stage, resulting in cascading discontinuities and the lack of an active uncertainty convergence mechanism.

[0005] According to a first aspect of this application, a method for interaction intent recognition and risk prediction that integrates multimodal perception is provided, comprising: S1, extract the temporal features of each modality from the acquired multimodal sensing signals of the operator; S2, the temporal features are processed using the evidence coding network corresponding to each modality, and a non-negative evidence vector corresponding to the preset interaction intention category is output. A Dirichlet distribution is constructed based on the non-negative evidence vector. The intention category probability of the modality is obtained from the expectation of the Dirichlet distribution, and the cognitive uncertainty of the modality is calculated from the sum of the lumped parameters of the Dirichlet distribution. S3, fuse the non-negative evidence vectors of each modality in the evidence space to generate joint evidence, determine the joint intention distribution and joint cognitive uncertainty based on the joint evidence, and calculate the conflict degree between the evidence of each modality, and take the joint intention distribution, joint cognitive uncertainty and conflict degree together as the uncertainty state; S4, input the uncertainty state and the scenario context information of the current human-machine shared control task into the risk prediction model, and output the risk level by the risk prediction model. The risk level increases as the joint cognitive uncertainty or conflict degree increases. S5. When the joint cognitive uncertainty exceeds the preset tolerance limit, an intent confirmation request is initiated. After obtaining the intent category confirmed by the operator, an enhanced evidence vector with the dominant evidence quantity in the confirmed intent category is generated. The enhanced evidence vector is used as a new modality and is re-fused with the non-negative evidence vectors of the current modalities to obtain the updated joint intent distribution and the updated joint cognitive uncertainty.

[0006] This application addresses the issues of cascading failures and lack of proactive uncertainty convergence mechanisms in related technologies. By employing an evidence coding network to process temporal features of each modality to obtain non-negative evidence vectors and constructing a Dirichlet distribution to simultaneously output intent category probabilities and cognitive uncertainty, multimodal evidence is fused in the evidence space to generate an uncertainty state containing joint intent distribution, joint cognitive uncertainty, and conflict degree. This uncertainty state is then combined with the scenario context and input into a risk prediction model to dynamically adjust the risk level with uncertainty. Furthermore, when joint cognitive uncertainty exceeds the limit, enhanced evidence vectors are injected for re-fusion to converge the uncertainty. Therefore, this approach solves the problems of cascading failures caused by the inability of intent recognition uncertainty to be transmitted to the risk prediction stage under a serial processing architecture in related technologies. It achieves the technical effects of fully transmitting the reliability information of intent reasoning, realizing dynamic risk adaptation assessment, and proactively and quickly converging cognitive uncertainty, thereby improving the accuracy of intent recognition and the security of risk decision-making in human-machine shared control scenarios.

[0007] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0008] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating a method for interaction intent recognition and risk prediction that integrates multimodal perception, as provided in an embodiment of this application. Detailed Implementation

[0010] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0011] The following describes, with reference to the accompanying drawings, an embodiment of the interactive intent recognition and risk prediction method integrating multimodal perception according to this application.

[0012] Figure 1 This is a flowchart illustrating a method for interaction intent recognition and risk prediction that integrates multimodal perception, as provided in an embodiment of this application.

[0013] like Figure 1 As shown, the method includes the following steps: S1 extracts the temporal features of each modality from the multimodal sensing signals collected from the operator.

[0014] In some embodiments, multimodal perception signals are acquired from the operator in a human-machine shared control scenario, including at least two of the following: electroencephalogram (EEG) signals, eye-tracking signals, and force signals. EEG signals are acquired through a wearable acquisition device and can reflect the operator's neural intention state; eye-tracking signals are acquired through an eye-tracking device and can provide the operator's attentional spatial positioning information; force signals are acquired through a force sensing unit and are used to characterize the execution state of the operator's limb movements. These multimodal signals carry the operator's interactive intention information from different dimensions, complementing and supporting each other.

[0015] The raw acquired signals for each modality are segmented according to fixed-duration time windows, and temporal features of the corresponding modality are extracted within each time window. These temporal features characterize the dynamic changes of the corresponding modal signal over time. Different feature extraction logics are adapted for different modalities to fully preserve intent-related information in the signal. The extracted temporal features for each modality have a unified temporal granularity, adapting to the input requirements of subsequent processing.

[0016] By extracting the temporal features of multi-dimensional sensory signals, the operator's intent-related information can be captured from multiple physiological and behavioral dimensions. The multimodal information complements each other and can maintain the validity of the input information when the single-modal signal fluctuates, providing a stable feature basis for subsequent intent reasoning.

[0017] S2, the temporal features are processed using the evidence encoding network corresponding to each modality, and a non-negative evidence vector corresponding to the preset interaction intent category is output. A Dirichlet distribution is constructed based on the non-negative evidence vector. The probability of the intent category of the modality is obtained from the expectation of the Dirichlet distribution, and the cognitive uncertainty of the modality is calculated from the sum of the lumped parameters of the Dirichlet distribution.

[0018] In some embodiments, a corresponding evidence encoding network is configured for each modality. The input layer dimension of each evidence encoding network is adapted to the temporal feature dimension of the corresponding modality. The main body of the network adopts a learnable temporal encoding structure to map the input temporal features into non-negative evidence vectors corresponding to preset interaction intent categories. Each element in the non-negative evidence vector has a non-negative value, representing the amount of evidence support for the corresponding intent category by the modality. The higher the element value, the stronger the support of the modality for the corresponding intent category. The total number of preset interaction intent categories can be set according to the specific human-machine shared control task. Taking the collaborative robot assembly task as an example, it can be set to 5 categories, corresponding to movement, stop, grasp, release, and emergency stop.

[0019] The Dirichlet distribution for the corresponding mode is constructed based on the output non-negative evidence vector. Specifically, each element of the non-negative evidence vector is incremented by one to obtain the lumped parameters of the Dirichlet distribution, calculated using the following formula:

[0020] in, Let be the element value corresponding to the j-th intent category in the non-negative evidence vector of the m-th modality. For the corresponding position, j is a set parameter, and its value ranges from 1 to the total number of preset interaction intent categories K.

[0021] Summing all the lumped parameters of this mode yields the sum of the lumped parameters. The calculation formula is:

[0022] The sum of lumped parameters reflects the total amount of evidence for that modality, when When the value is only slightly greater than K, the Dirichlet distribution tends to be flat and uniform, indicating that this mode lacks a clear criterion for judgment; when When the value is significantly greater than K, the distribution tends to be sharp and concentrated, indicating that the basis for judging this mode is sufficient.

[0023] Based on the constructed Dirichlet distribution, the intent category probability and cognitive uncertainty of this modality are calculated simultaneously. The intent category probability is given by the expectation of the Dirichlet distribution, and the prediction probability of the m-th modality for the j-th intent category is:

[0024] The sum of the probability values ​​of all intent categories is 1, which together constitutes the intent category probability distribution for this modality.

[0025] Cognitive uncertainty is calculated as the ratio of the total number of preset interaction intent categories to the sum of set parameters, using the following formula:

[0026] Cognitive uncertainty decreases monotonically with increasing total evidence, and can quantify the reliability of the modality's intent judgment. In actual operation scenarios, when the operator performs a skilled grasping action, the modal signal is clear and consistent, and the element values ​​of the output non-negative evidence vector at the corresponding intent category position are much larger than those at other positions, with the sum of lumped parameters reaching tens, and cognitive uncertainty remains between 0.1 and 0.3. When the operator is hesitant about the next action, the signal features are blurred, the element values ​​of the non-negative evidence vector are generally low and evenly distributed, the sum of lumped parameters is close to the total number of categories K, and cognitive uncertainty can rise to over 0.6.

[0027] By encoding temporal features through an evidence coding network and constructing a Dirichlet distribution based on non-negative evidence vectors, the system can simultaneously output the probability of intent categories and the quantified cognitive uncertainty in the single-modal intent recognition stage. This enables the simultaneous output of intent judgment results and reliability, providing complete probability and uncertainty information support for subsequent multimodal fusion and risk prediction stages.

[0028] S3, fuse the non-negative evidence vectors of each modality in the evidence space to generate joint evidence, determine the joint intention distribution and joint cognitive uncertainty based on the joint evidence, and calculate the conflict degree between the evidence of each modality, and take the joint intention distribution, joint cognitive uncertainty and conflict degree as the uncertainty state.

[0029] In some embodiments, after generating non-negative evidence vectors for each modality, a fusion process is performed on all non-negative evidence vectors in the evidence space to generate joint evidence that integrates information from all modalities. Evidence space fusion directly combines the intent support information of each modality in the dimension of evidence quantity. Compared with traditional probabilistic fusion methods, it can completely preserve the evidence strength and uncertainty information of each modality and avoid unnecessary loss of reliability information during the fusion process.

[0030] Based on the generated joint evidence, the joint intent distribution and joint cognitive uncertainty are determined. Joint evidence includes non-negative evidence values ​​corresponding one-to-one with preset interaction intent categories. Following the construction logic of a unimodal Dirichlet distribution, each element of the joint evidence is incremented by one to obtain joint set parameters. The sum of all joint set parameters is then obtained. The joint intent distribution is determined by the ratio of each joint set parameter to the sum of the joint set parameters. The value of each category corresponds to the probability of occurrence of that intent category after fusion. The sum of the probabilities of all categories is 1, intuitively reflecting the overall intent judgment tendency after multimodal information fusion. Joint cognitive uncertainty is calculated by the ratio of the total number of preset interaction intent categories to the sum of the joint set parameters. It is used to quantify the unreliability of the overall intent judgment after multimodal fusion; a higher value indicates less sufficient evidence for the fused judgment. In typical operational scenarios where the signals of each modality are clear and consistent, the joint cognitive uncertainty after fusion can be as low as approximately 0.15, indicating a relatively high level of judgment reliability.

[0031] The conflict degree between different modal evidences is calculated simultaneously. The conflict degree measures the degree of divergence between the evidence from different modalities, directly reflecting the consistency level of the multimodal signals in intent judgment. The more unified the evidence is between modalities, the lower the conflict degree value; the more obvious the contradiction between the evidence is between modalities, the higher the conflict degree value. In scenarios where the signals of each modality are highly consistent, the conflict degree can be maintained at a low level of around 0.1.

[0032] The joint intent distribution, joint cognitive uncertainty, and conflict degree are collectively encapsulated into an uncertainty state, which is then passed to downstream processes as the final output of this step. The system outputs the corresponding uncertainty state according to a fixed time window, fully carrying three core types of information: intent tendency, overall reliability, and modal divergence degree after multimodal fusion. This replaces the single deterministic intent label in traditional solutions, providing a comprehensive decision-making basis for downstream risk assessment and uncertainty control.

[0033] By fusing multimodal evidence in the evidence space, a multidimensional uncertainty state containing joint intent distribution, joint cognitive uncertainty, and conflict degree is generated simultaneously. This can fully preserve the reliability of intent judgment and modal divergence information while integrating multi-source perception information, providing comprehensive uncertainty input for downstream risk prediction and effectively solving the risk assessment cascade discontinuity problem caused by the loss of uncertainty information in the traditional serial architecture.

[0034] S4, input the uncertainty state and the scenario context information of the current human-machine shared control task into the risk prediction model, and output the risk level by the risk prediction model. The risk level increases as the joint cognitive uncertainty or conflict degree increases.

[0035] In some embodiments, after acquiring the uncertainty state output from the upstream, it is combined with the scenario context information of the current human-machine shared control task and jointly input into the risk prediction model. The scenario context information corresponds to the real-time operating environment and operational state parameters of the current task. In a collaborative robot assembly scenario, it may include real-time collected data such as the distance between the operator and the robot's end effector, the robot's end effector speed, and the distance to the nearest obstacle. The original scenario context information is first processed by feature encoding and mapped to a fixed-dimensional context encoding vector through a fully connected layer. For example, the three types of numerical context parameters are mapped to an 8-dimensional feature vector to achieve dimensional adaptation with the uncertainty state data.

[0036] The risk prediction model employs a temporal neural network architecture to capture the temporal evolution of risk states. The system maintains a fixed-length sliding window to store the composite input vector for the current moment and several consecutive previous moments. Each moment's composite input vector is obtained by concatenating the joint intent distribution probability value, joint cognitive uncertainty, and conflict degree in the uncertain state with the corresponding moment's scene context encoding vector. When the preset total number of interaction intent categories is 5, the joint intent distribution contains 5 probability values. After superimposing the joint cognitive uncertainty, conflict degree, and 8-dimensional scene context encoding, the total dimension of the composite input vector is 15. The composite input vector is first mapped to a 64-dimensional embedding vector through a linear projection layer, then sine-cosine position encoding is used to add temporal position information to the sequence, followed by a 3-layer Transformer encoder for temporal feature extraction. Finally, the output layer yields the risk level result. The sliding window length can be set to 10, covering historical data for 10 consecutive moments to ensure the temporal continuity of risk assessment.

[0037] The risk level output by the model is directly related to the state of uncertainty. The risk level increases with the increase of joint cognitive uncertainty, and also with the increase of intermodal conflict. The unreliability of intent judgment itself participates in risk assessment as a safety signal. When the system's confidence in the operator's intent is low, even if the intent with the highest probability has a low risk level in normal scenarios, the risk level will be adjusted upward accordingly. This allows risk prediction to fully perceive the reliability of upstream intent reasoning and avoids the bias caused by assessing risk based solely on a single intent label.

[0038] By inputting multi-dimensional uncertainty states and scenario context into the risk prediction model, the reliability of risk level arbitrary graph judgment is dynamically adjusted, the transmission link from uncertainty in intent recognition to risk prediction is opened up, and the problem of risk assessment cascade discontinuity caused by the loss of uncertainty information in traditional serial architecture is solved, effectively improving the rationality and security of risk decision-making in human-machine shared control scenarios.

[0039] S5. When the joint cognitive uncertainty exceeds the preset tolerance limit, an intent confirmation request is initiated. After obtaining the intent category confirmed by the operator, an enhanced evidence vector with the dominant evidence quantity in the confirmed intent category is generated. The enhanced evidence vector is used as a new modality and is re-fused with the non-negative evidence vectors of the current modalities to obtain the updated joint intent distribution and the updated joint cognitive uncertainty.

[0040] In some embodiments, the system tracks the joint cognitive uncertainty output by multimodal evidence fusion in real time and continuously compares it with a preset tolerance limit. The tolerance limit is preset based on the safety requirements of the human-machine shared control task and represents the lower limit of the reliability of the intent judgment for which the system can normally perform risk decisions. When the joint cognitive uncertainty exceeds this tolerance limit, it indicates that the current multimodal perception information alone cannot form a sufficiently reliable intent judgment result. The system then initiates an intent confirmation request, introducing explicit confirmation information from the operator through proactive human-machine interaction.

[0041] After obtaining the operator's confirmed intent category, the system generates an enhanced evidence vector. This enhanced evidence vector possesses dominant evidence for the confirmed intent category, giving that category significantly higher evidence support than other categories. The system treats this enhanced evidence vector as a new independent modality and inputs it, along with the non-negative evidence vectors from each of the existing modalities, into the evidence space fusion process to re-execute the multimodal evidence fusion operation.

[0042] After re-fusion, the system outputs an updated joint intention distribution and an updated joint cognitive uncertainty. The operator's confirmation information, injected into the evidence space as high-certainty evidence, quickly guides the fused belief distribution towards the confirmation category, causing the joint cognitive uncertainty to rapidly fall below the tolerance limit. This propels the system from a state of high uncertainty back to a stable and reliable state of intention judgment and risk decision-making, forming a cognitive closed loop that actively converges uncertainty.

[0043] By using an active confirmation and evidence re-fusion mechanism triggered by excessive uncertainty, when multimodal perception information is insufficient to support reliable judgment, highly certain evidence confirmed by humans can be introduced to quickly converge cognitive uncertainty, construct an active control intention recognition closed loop, and effectively improve the reliability of intention judgment and the security of human-computer interaction in complex scenarios.

[0044] Compared with related technologies, in this embodiment, temporal features of each modality are extracted from the collected multimodal perception signals of the operator; the temporal features are processed using the evidence coding network corresponding to each modality to output non-negative evidence vectors corresponding to preset interaction intent categories; a Dirichlet distribution is constructed based on the non-negative evidence vectors; the probability of the intent category of the modality is obtained from the expectation of the Dirichlet distribution; and the cognitive uncertainty of the modality is calculated from the sum of the lumped parameters of the Dirichlet distribution; the non-negative evidence vectors of each modality are fused in the evidence space to generate joint evidence; the joint intent distribution and joint cognitive uncertainty are determined based on the joint evidence; and the conflict degree between the evidence of each modality is calculated. The uncertainty state is determined by combining cognitive uncertainty and conflict degree. This uncertainty state, along with the context information of the current human-machine shared control task, is input into a risk prediction model. The model outputs a risk level, which increases with the increase of the joint cognitive uncertainty or conflict degree. When the joint cognitive uncertainty exceeds a preset tolerance limit, an intent confirmation request is initiated. After obtaining the intent category confirmed by the operator, an enhanced evidence vector with dominant evidence quantity in the confirmed intent category is generated. This enhanced evidence vector is then used as a new modality and re-fused with the non-negative evidence vectors of the current modalities to obtain an updated joint intent distribution and an updated joint cognitive uncertainty. This approach addresses the cascading breakdown caused by the inability to transmit intent recognition uncertainty to the risk prediction stage in a serial processing architecture, and the lack of an active uncertainty convergence mechanism. It achieves the technical effects of fully transmitting the reliability information of intent reasoning, realizing dynamic risk adaptation assessment, actively and quickly converging cognitive uncertainty, and improving the accuracy of intent recognition and the security of risk decision-making in human-machine shared control scenarios.

[0045] As a specific implementation of this application, based on the basic scheme, the cognitive uncertainty in S2 is further defined as the ratio of the total number of preset interaction intention categories to the sum of the set parameters, and the set parameters are obtained by adding one to each element of the non-negative evidence vector.

[0046] Specifically, in this embodiment, the quantitative calculation of single-modal cognitive uncertainty is specifically limited, and an uncertainty metric with clear physical meaning and monotonic change law is obtained through standardized calculation logic.

[0047] First, the lumped parameters of the Dirichlet distribution are constructed based on the non-negative evidence vectors. For the non-negative evidence vector of the m-th modality output, the lumped parameters at each corresponding position are incremented by one, and the calculation formula is as follows:

[0048] in, Let be the element value corresponding to the j-th intent category in the non-negative evidence vector of the m-th modality. For the corresponding position, j is a set parameter, and its value ranges from 1 to the total number of preset interaction intent categories K.

[0049] The sum of the lumped parameters for all categories of this mode is obtained by summing the lumped parameters of this mode. The calculation formula is as follows:

[0050] Where K is the total number of preset interaction intent categories, and the sum of the parameters represents the total amount of evidence that the modality can be used for intent judgment.

[0051] Ultimately, the cognitive uncertainty of this modality is calculated as the ratio of the total number of preset interaction intent categories to the sum of lumped parameters, using the following formula:

[0052] This calculation method exhibits a clear pattern of change: when a modality completely lacks valid judgment criteria, all elements of the non-negative evidence vector are 0, and the sum of the lumped parameters equals the total number of categories K. At this point, the cognitive uncertainty reaches its maximum value of 1. As the number of valid evidence collected for a modality increases, the sum of the lumped parameters continues to increase, and the cognitive uncertainty monotonically decreases, eventually approaching 0. The uncertainty value output by the entire calculation process is constrained within a standardized range of 0 to 1, and can be directly used in subsequent multimodal fusion and risk assessment stages.

[0053] In actual operation scenarios, when the operator performs skilled operations, the signal features of the corresponding modality are clear and distinct. The element values ​​of the non-negative evidence vector in the target intent category are significantly higher than those in other categories, and the sum of the lumped parameters can reach tens. The corresponding cognitive uncertainty is maintained between 0.1 and 0.3, indicating that the intent judgment has a high degree of reliability. When the operator is hesitant about the operation and the signal features are ambiguous, the element values ​​of the non-negative evidence vector are generally low and evenly distributed. The sum of the lumped parameters is close to the total number of categories K, and the cognitive uncertainty can rise to more than 0.6, which can sensitively reflect the lack of evidence for the current intent judgment.

[0054] This standardized method for calculating cognitive uncertainty yields quantitative results of uncertainty with clear physical meaning and monotonous variation. It can accurately and sensitively reflect the sufficiency of evidence for single-modal intent judgment, providing a unified and reliable basis for subsequent multimodal evidence fusion and risk level assessment.

[0055] As a specific implementation of this application, based on the basic scheme, the S3 step is further defined as follows: fusing the non-negative evidence vectors of each modality in the evidence space, including: The joint evidence is generated by performing an orthogonal sum operation on the non-negative evidence vectors of each modality using the Dempster combination rule.

[0056] Specifically, in this embodiment, the fusion of multimodal nonnegative evidence vectors in the evidence space adopts the Dempster combination rule, which uses orthogonal summation to standardize and aggregate the evidence information of each modality to generate joint evidence.

[0057] Before fusion, the non-negative evidence vectors of each modality are first transformed into basic belief assignments. For a single modality, the basic belief corresponding to each intention category is the amount of evidence for that category divided by the sum of the lumped parameters of that modality; at the same time, the remaining amount of evidence is assigned to the entire set, and the beliefs in the entire set correspond to the amount of evidence that the modality cannot determine to belong to any specific category, carrying the cognitive uncertainty information of the modality itself.

[0058] Subsequently, an orthogonal summation operation is performed on the belief assignments between modalities. For any intention category, the joint belief consists of three contributions: the product of beliefs from both modalities pointing to that category, the product of beliefs from one modality pointing to that category while the other modality's belief remains in the entire set, and the symmetric reverse case. Summing these three components and dividing by the normalization factor N yields the joint belief for that category. The normalization factor N equals 1 minus the sum of all products of beliefs supporting different categories, representing the total amount of non-conflicting components in the two modal beliefs.

[0059] After obtaining the joint beliefs for all intent categories, the joint evidence vector is derived from the joint beliefs and used as the joint evidence output for multimodal fusion. When the system contains only two modalities, the fusion can be completed by directly executing the above single Dempster combination; when it contains three or more modalities, a pairwise recursive approach is used to combine them sequentially. First, the evidence fusion of the first two modalities is completed, and then the obtained joint evidence is combined with the third modality to continue to execute the Dempster combination, until the evidence aggregation of all modalities is completed.

[0060] By employing the Dempster combination rule to perform orthogonalization and fusion of evidence space, multi-source modal evidence can be standardized and integrated based on the evidence theory system. During the fusion process, the strength of evidence and uncertainty components are fully preserved, and multi-dimensional intent information is effectively aggregated. This ensures the rationality and accuracy of joint evidence and provides a reliable foundation for the subsequent calculation of joint uncertainty states.

[0061] As a specific implementation of this application, the degree of conflict is further limited based on the basic solution by one of the following methods: Calculate the symmetric divergence between the Dirichlet distributions corresponding to different modes; Alternatively, calculate the standardized conflict coefficient in the Dempster combination process.

[0062] Specifically, this implementation provides two optional methods for calculating the conflict degree between modal evidence, which can be flexibly selected according to the computing power conditions and accuracy requirements of the actual application scenario.

[0063] The first method calculates the degree of conflict by measuring the symmetric divergence between the Dirichlet distributions corresponding to different modalities. Symmetric divergence quantifies the degree of difference between two probability distributions; a larger divergence value indicates a greater deviation in the intended judgments of the two modalities, and a more significant discrepancy in the evidence. When the system contains three or more modalities, the symmetric divergence of the Dirichlet distributions corresponding to each pair of modalities is calculated sequentially, and the maximum value among all divergence values ​​is taken as the final degree of conflict, thus reflecting the overall level of disagreement among the multimodal evidence.

[0064] The second approach uses the standardized conflict coefficient from the Dempster combination process as the conflict degree, calculated as δ=1-N, where N is the normalization factor in the Dempster combination operation, representing the total amount of non-conflicting components in the two modal beliefs. The conflict degree obtained using this method ranges from 0 to 1; when the evidence from the two modalities is highly consistent, the conflict degree approaches 0, and when they are highly contradictory, the conflict degree approaches 1, possessing clear physical meaning and a standardized value range.

[0065] By providing two differentiated conflict degree calculation paths, it can adapt to different system deployment conditions and application requirements. Both paths can accurately quantify the degree of disagreement among multimodal evidence, providing reliable modal consistency measurement support for the construction of uncertain states and downstream risk prediction.

[0066] As a specific implementation of this application, based on the basic scheme, it further includes, before fusion using the aforementioned Dempster combination rule: Monitor the cognitive uncertainty of each modality, or the level of conflict between the evidence vectors of each modality and the existing joint evidence; When the cognitive uncertainty or conflict level of any modality exceeds a set threshold, the non-negative evidence vector of that modality is multiplied by a decay factor to obtain a decayed evidence vector, which is then used during fusion.

[0067] Specifically, before performing multimodal evidence fusion using the Dempster combination rule, this implementation adds an internal feedback attenuation control loop. By dynamically adjusting the evidence weights of low-quality modalities, the robustness of multimodal fusion to single-modal signal quality fluctuations is improved.

[0068] During the fusion execution phase of each time window, the system simultaneously monitors two indicators for each modality. The first is the cognitive uncertainty of the modality itself, directly taken from the output of the single-modality evidence encoding stage, reflecting the reliability of the current modality's independent intent judgment. The second is the conflict level between the modality and existing joint evidence. It is calculated by taking the existing joint intent distribution output from the previous time window and performing a symmetrical divergence calculation with the intent category probability distribution of the current modality. The larger the divergence value, the greater the deviation between the current judgment of the modality and the system's existing fusion judgment, and the more obvious the discrepancy in the evidence.

[0069] The preset cognitive uncertainty threshold is 0.5, and the conflict level threshold is 0.8. The system evaluates each modality one by one. If the cognitive uncertainty of a modality is greater than 0.5, or the conflict level is greater than 0.8, the non-negative evidence vector of that modality is multiplied by a decay factor of 0.3 to obtain a decayed evidence vector, which is then used in the subsequent Dempster combination operation. If neither of the two indicators exceeds the corresponding threshold, the original non-negative evidence vector is used directly in the fusion.

[0070] In real-world scenarios, if operator fatigue leads to a decline in EEG signal quality, the values ​​of each element in the non-negative evidence vector output by the EEG modality will generally be low, and cognitive uncertainty will rise to 0.7, exceeding the preset threshold of 0.5. Internal feedback attenuation control will automatically attenuate the evidence vector of the EEG modality, reducing its evidence weight to 0.3 times its original value. At this time, the force signal modality, because the operator is still applying force stably, maintains a cognitive uncertainty of 0.2, which does not meet the attenuation condition and can participate in fusion normally. The final multimodal fusion result will be primarily dominated by the reliable force signal modality, and the joint cognitive uncertainty will not increase drastically due to the temporary deterioration of the EEG signal. When the operator adjusts their state and the EEG signal quality recovers, their cognitive uncertainty falls back below 0.5, the attenuation process is automatically lifted, and the EEG modality regains its full weight and re-participates in evidence fusion.

[0071] By employing a dynamic attenuation mechanism based on modal cognitive uncertainty and conflict levels, evidence contributions from modalities with low reliability or significant deviations from the overall judgment can be automatically suppressed, reducing interference from unreliable signal sources on the fusion results. This ensures that joint cognitive uncertainty remains relatively stable even when the quality of individual modal signals declines, effectively enhancing the robustness of multimodal evidence fusion and the continuity and stability of intent judgment.

[0072] As a specific implementation of this application, based on the basic solution, the risk prediction model in S4 is further defined as a temporal neural network. The input to the risk prediction model is a concatenated vector of the uncertain states and scene context information at multiple consecutive time points within a sliding window. The output is the probability distribution of risk levels for the current and future preset time periods.

[0073] Specifically, in this embodiment, the risk prediction model adopts a temporal neural network architecture, relying on the ability to extract temporal features to capture the continuous evolution of risk status during human-computer interaction, thereby achieving dynamic temporal prediction of risk level.

[0074] The system maintains a fixed-length sliding window to store historical operational data for the current moment and several consecutive moments prior. The sliding window length can be set to 10, covering the state information for 10 consecutive moments. The input data for each moment is a composite vector formed by concatenating the uncertain state of that moment with the scene context information. The uncertain state includes all probability values ​​of the joint intent distribution, the joint cognitive uncertainty value, and the conflict degree value. The scene context information is first encoded into a fixed-dimensional feature vector through a fully connected layer before being concatenated. When the preset total number of interaction intent categories is 5 and the scene context encoding is 8-dimensional, the total dimension of the composite input vector at a single moment is 15.

[0075] The composite input vector is first mapped to a higher-dimensional embedding vector through a linear projection layer, such as a 64-dimensional embedding vector. Then, sine-cosine position encoding is used to add temporal position information to the sequence. Subsequently, it is fed into a multi-layer Transformer encoder for deep temporal feature extraction. Finally, the model output layer generates the prediction result.

[0076] The model outputs a probability distribution of risk levels for the current and future preset time periods. The results not only include the probability of occurrence of each risk level at the current moment, but also cover the risk evolution trend within the preset time period, which can provide a more forward-looking decision-making basis for the formulation of downstream security strategies.

[0077] A risk prediction model is constructed using a temporal neural network. Based on multi-time historical data from a sliding window, the model completes the temporal modeling of risks and outputs a probability distribution of risk levels covering the current and future periods. This model can more accurately capture the dynamic changes in risk status, improve the foresight and accuracy of risk prediction, and provide reliable support for safety decision-making in human-machine shared control scenarios.

[0078] As a specific implementation of this application, based on the basic scheme, the loss function used when training the risk prediction model is further limited to include an uncertainty-sensitive penalty term; When the joint cognitive uncertainty is greater than the first threshold or the conflict degree is greater than the second threshold, and the expected risk level output by the risk prediction model is less than the third threshold, the uncertainty-sensitive penalty term is a positive penalty value; otherwise, it is zero.

[0079] Specifically, this implementation optimizes the training process of the risk prediction model by introducing an uncertainty-sensitive penalty term into the loss function. It strengthens the model's risk response characteristics to high-uncertainty scenarios through targeted constraints, ensuring that the trained model conforms to the safety design logic that the risk level increases with the increase of uncertainty.

[0080] The loss function consists of a base classification loss and an uncertainty-sensitive penalty term. The base classification loss constrains the degree of matching between the risk level prediction and the ground truth label, while the uncertainty-sensitive penalty term corrects the model's risk underestimation behavior in high-uncertainty scenarios. The uncertainty-sensitive penalty term employs a conditional triggering mechanism, taking a positive penalty value only when both conditions are met simultaneously; otherwise, it takes a value of zero, thus not interfering with the training process in normal scenarios.

[0081] The first trigger condition is that the uncertainty state corresponding to the input sample meets the high uncertainty criterion: the joint cognitive uncertainty is greater than a preset first threshold, or the intermodal conflict degree is greater than a preset second threshold, indicating that the reliability of the current upstream intention judgment is insufficient, and according to the safety design principle, a higher risk assessment result should be given. The second trigger condition is that the expected risk level output by the model for this sample is less than a preset third threshold, indicating that the model has given a low risk assessment result in a high uncertainty scenario, which is inconsistent with the design expectation. When both conditions are met, the uncertainty-sensitive penalty term takes a preset positive penalty value and is added to the total loss to participate in backpropagation, guiding the model to adjust parameters and gradually improve the risk output level in high uncertainty scenarios; if either condition is not met, the penalty term takes a value of zero, and the loss function is dominated only by the basic classification loss, ensuring the prediction accuracy in normal scenarios.

[0082] By introducing a conditionally triggered uncertainty-sensitive penalty term into the loss function, the risk output logic of the model can be directionally constrained during the training phase. This ensures a positive correlation between the model's learned intent uncertainty and the risk level, preventing the model from underestimating risk in scenarios with high cognitive uncertainty or high modal conflict, and effectively enhancing the safety orientation and reliability of the risk prediction model.

[0083] As a specific implementation of this application, based on the basic scheme, the step of S5 is further defined as generating an enhanced evidence vector with a dominant amount of evidence in the confirmed intent category, including: Set the evidence element corresponding to the confirmed intent category as a first value, and set the evidence elements of the other categories as a second value, wherein the first value is at least 10 times the second value.

[0084] Specifically, this implementation method specifies the generation rules for the enhanced evidence vector in step S5. By configuring a significantly different amount of evidence, it ensures that the intention category confirmed by the operator receives dominant belief support during the re-fusion process, thus ensuring that cognitive uncertainty can converge quickly.

[0085] When generating the enhanced evidence vector, the evidence element corresponding to the intent category confirmed by the operator is set as the first value, and the evidence elements corresponding to all other preset interaction intent categories are set as the second value. The first value is at least 10 times the second value. This multiple difference gives the confirmed category an absolute evidentiary advantage, and the evidence amount of the other categories is insufficient to significantly interfere with the overall judgment tendency after fusion. When this enhanced evidence vector participates in the evidence space refusion as a new modality, the high evidence amount of the confirmed category will quickly increase the joint evidence support of the corresponding category, guiding the joint intent distribution to concentrate on the confirmed category, and causing the joint cognitive uncertainty to quickly fall back to within the preset tolerance limit. In practical applications, the second value can be set to 1, corresponding to a first value of 10 or above, which can achieve effective convergence of uncertainty in a single refusion process.

[0086] By clearly defining the multiple of evidence, it can be ensured that the operator's confirmed information plays a dominant role as highly certain evidence, effectively improving the efficiency and stability of uncertainty convergence and guaranteeing the regulatory effect of the proactive cognition closed loop.

[0087] As a specific implementation of this application, based on the basic solution, it is further limited that when an intent confirmation request is initiated and no operator confirmation of the intent category is obtained within a preset waiting time, the current risk level is maintained or a preset security policy is executed.

[0088] Specifically, this implementation method sets up a safety fallback mechanism for timeout scenarios of intent confirmation requests to avoid introducing unreliable speculative information due to the operator's failure to respond in a timely manner, thus ensuring the safety and controllability of the human-computer interaction process.

[0089] After the system initiates an intent confirmation request, it simultaneously starts a timer and continuously monitors the operator's feedback input. If the operator's confirmed intent category is successfully obtained within the preset waiting time, the subsequent enhanced evidence vector generation and evidence space re-fusion process proceed normally. If no valid operator confirmation information is obtained by the end of the preset waiting time, no speculative judgment is made regarding the operator's intent, and one of two types of safety processing logic is directly executed. One type maintains the current risk level, ensuring the existing risk control intensity does not decrease, and avoiding unfounded reductions in the risk level that could introduce safety hazards. The other type executes a preset safety strategy to further strengthen the safety control level. For example, in a collaborative robot assembly scenario, the robot's end effector speed is reduced to 20% of the preset safe speed limit while maintaining its current pose, awaiting subsequent explicit instructions from the operator.

[0090] By implementing safety fallback measures in timeout scenarios, unreliable speculative information can be prevented from entering the intent judgment and risk decision-making process when the operator cannot confirm the intent in a timely manner. This maintains a high level of secure interaction control and effectively enhances the fault tolerance and security level of human-machine shared control scenarios.

[0091] As a specific implementation of this application, based on the basic scheme, the multimodal sensing signal is further defined to include at least two of the following: electroencephalogram (EEG) signals, eye movement signals, and manipulative force signals.

[0092] Specifically, this embodiment defines the composition of the multimodal sensing signals. The sensing signals are selected from at least two of the following: electroencephalogram (EEG) signals, eye movement signals, and manipulative force signals. The operator's state information is collected from different physiological and behavioral dimensions to provide multi-source complementary input basis for intent recognition.

[0093] Electroencephalogram (EEG) signals, acquired through wearable devices, reflect the operator's neural intention state and can capture deep intention information such as preparatory potentials in advance, offering a time advantage in intention prediction. Eye-tracking signals, acquired through eye-tracking devices, provide spatial positioning information of the operator's attention, intuitively reflecting the target area of ​​visual focus and the direction of operation, providing a basis for spatial positioning of intention. Force signals, acquired through force sensing units, characterize the execution state of the operator's limb movements, directly reflecting changes in the force and trend of the actual operation, and reflecting the externalized operational behavior.

[0094] In practical applications, modal combinations can be flexibly selected based on the hardware conditions and task requirements of the deployment scenario. When deployment conditions are complete, signals from three modalities can be collected simultaneously, achieving full-dimensional information coverage of the neural, perceptual, and executive layers. If the application scenario does not have eye-tracking acquisition capabilities, only EEG signals and manipulative force signals can be used, still forming multi-dimensional information complementarity. The combination of multiple modalities can provide reliable information support from other modalities when the quality of a single modal signal fluctuates, avoiding significant deviations in intent judgment due to the failure of a single signal.

[0095] By selecting at least two different dimensions of perceptual signals to construct multimodal inputs, a perceptual system with complementary information can be formed, effectively reducing the impact of single-modal signal fluctuations on intent recognition, improving the robustness of the overall perceptual system, and providing a stable, multi-dimensional information foundation for subsequent intent reasoning.

[0096] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for interaction intent recognition and risk prediction integrating multimodal perception, characterized in that, include: S1, extract the temporal features of each modality from the acquired multimodal sensing signals of the operator; S2, the temporal features are processed using the evidence coding network corresponding to each modality, and a non-negative evidence vector corresponding to the preset interaction intention category is output. A Dirichlet distribution is constructed based on the non-negative evidence vector. The intention category probability of the modality is obtained from the expectation of the Dirichlet distribution, and the cognitive uncertainty of the modality is calculated from the sum of the lumped parameters of the Dirichlet distribution. S3, fuse the non-negative evidence vectors of each modality in the evidence space to generate joint evidence, determine the joint intention distribution and joint cognitive uncertainty based on the joint evidence, and calculate the conflict degree between the evidence of each modality, and take the joint intention distribution, joint cognitive uncertainty and conflict degree together as the uncertainty state; S4, input the uncertainty state and the scenario context information of the current human-machine shared control task into the risk prediction model, and output the risk level by the risk prediction model. The risk level increases as the joint cognitive uncertainty or conflict degree increases. S5. When the joint cognitive uncertainty exceeds the preset tolerance limit, an intent confirmation request is initiated. After obtaining the intent category confirmed by the operator, an enhanced evidence vector with the dominant evidence quantity in the confirmed intent category is generated. The enhanced evidence vector is used as a new modality and is re-fused with the non-negative evidence vectors of the current modalities to obtain the updated joint intent distribution and the updated joint cognitive uncertainty.

2. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 1, characterized in that, In S2, the cognitive uncertainty is the ratio of the total number of preset interaction intent categories to the sum of the set parameters, which are obtained by adding one to each element of the non-negative evidence vector.

3. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 1, characterized in that, In step S3, the non-negative evidence vectors of each modality are fused in the evidence space, including: The joint evidence is generated by performing an orthogonal sum operation on the non-negative evidence vectors of each modality using the Dempster combination rule.

4. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 3, characterized in that, The degree of conflict is obtained through one of the following methods: Calculate the symmetric divergence between the Dirichlet distributions corresponding to different modes; Alternatively, calculate the standardized conflict coefficient in the Dempster combination process.

5. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 3, characterized in that, Before fusion using the Dempster combination rule, the following is also included: Monitor the cognitive uncertainty of each modality, or the level of conflict between the evidence vectors of each modality and the existing joint evidence; When the cognitive uncertainty or conflict level of any modality exceeds a set threshold, the non-negative evidence vector of that modality is multiplied by a decay factor to obtain a decayed evidence vector, which is then used during fusion.

6. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 1, characterized in that, In step S4, the risk prediction model is a time-series neural network. The input to the risk prediction model is a concatenated vector of the uncertain states and scene context information at multiple consecutive time points within a sliding window. The output is the probability distribution of risk levels for the current and future preset time periods.

7. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 6, characterized in that, The loss function used during the training of the risk prediction model includes an uncertainty-sensitive penalty term; When the joint cognitive uncertainty is greater than the first threshold or the conflict degree is greater than the second threshold, and the expected risk level output by the risk prediction model is less than the third threshold, the uncertainty-sensitive penalty term is a positive penalty value; otherwise, it is zero.

8. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 1, characterized in that, In step S5, generating an enhanced evidence vector with dominant evidence quantity for the confirmed intent category includes: Set the evidence element corresponding to the confirmed intent category as a first value, and set the evidence elements of the other categories as a second value, wherein the first value is at least 10 times the second value.

9. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 8, characterized in that, If an intent confirmation request is initiated and no operator confirmation of the intent category is obtained within the preset waiting time, the current risk level will be maintained or the preset security policy will be executed.

10. The interactive intent recognition and risk prediction method integrating multimodal perception according to claim 1, characterized in that, The multimodal sensing signals include at least two of the following: electroencephalogram (EEG) signals, eye movement signals, and manipulative force signals.

Images