A human-robot bidirectional cognitive security interaction method

Abstract

The application discloses a kind of human and robot bidirectional cognitive safety interaction method, based on the framework of augmented reality and physiological signal, a kind of physiological signal driven augmented reality in human and robot bidirectional cognitive safety interaction strategy is presented.One aspect through sensor real-time collection of human physiological signal, and construct a physiological data driven perception prediction model, for the average safety index of human perception, robot is adapted according to the average safety index and adjusts behavior, so that the cognition of robot to human state (perception safety etc.) is enhanced, on the other hand, the cognition of human to robot intention and working environment is enhanced using three kinds of augmented reality technology.In this way, when carrying out the safety interaction such as " handover task " of human and robot, not only satisfy the safety design guide of robot behavior etc., but also can dynamically adjust robot behavior through bidirectional intention cognition, so as to guarantee the efficiency while constructing safe interaction space.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-robot cognitive interaction, and more specifically relates to a human-robot bidirectional cognitive safe interaction method. BACKGROUND

[0002] In recent years, collaborative robots have broken the traditional human-machine isolation working mode and share the working space with humans. Collaborative robots not only can improve the quality of life of the elderly at home, improve the learning participation of students, and improve office efficiency, but also can provide important technical support for various fields such as manufacturing, services, health care, and agriculture, such as handling and handover.

[0003] In order to achieve good human-robot (referred to as human-robot) collaboration, it is crucial to ensure the safety of humans. Collaborative robots must meet safety design guidelines such as ISO / TS 15066 to ensure the safety of humans and avoid accidental collisions. These guidelines not only apply to industrial environments, but also to daily life environments. In these environments, the tasks of collaborative robots may involve direct interaction with humans, such as handover (picking up and passing objects). When collaborative robots do not provide appropriate cues, human-robot handover may fail in various ways: humans may not be able to identify that the robot is giving them an object, the object may fall, or humans may feel uncomfortable or unsafe during the interaction.

[0004] In human-to-human interaction, gaze cues have been shown to effectively communicate attention. Inspired by human-to-human interaction, Moon implemented a gaze behavior (called "turn-to-gaze") for a humanoid robot and demonstrated that gaze cues can increase the naturalness of the handover process. However, the "gaze cues" of the robot can improve the naturalness of the human-robot handover process, but this single interaction cue may lead to: 1) unclear handover time (when people reach out, when robots release). "Turn-to-gaze" may distract humans, prolong handover time, or cause misjudgment of handover time. 2) unclear handover position. 3) unclear handover intention.

[0005] Currently, collaborative robots often lack effective intention expression capabilities, resulting in unsafe, unreliable, or smooth handover processes. Augmented reality (AR) technology can superimpose virtual information on the real world in the field of view of humans, which can enhance human perception and cognition of robot intentions (behaviors and states, etc.) and the environment, and solve the limitations of language, gestures, and other interaction methods (delay causes humans to miss the best handover opportunity, ambiguity causes humans to misunderstand the intentions of the robot, distracts humans, and is unnatural, etc.), thereby improving the safety and smoothness of the handover task.

[0006] Newbury used HoloLens 2 to display the robot's goals and plans, and experiments showed that AR can improve user experience, especially when the robot estimates errors, it can help users better correct errors, improve the smoothness and safety of handover. When augmented reality is introduced into the handover task, it is necessary to select an appropriate visualization method to clearly indicate the handover opportunity, location and intention. However, Newbury's system only enhances the one-way cognition of humans to robots, lacks the cognition of robots to humans (perception of safety, etc.), and hinders safe human-robot interaction under bidirectional cognition.

[0007] At present, in human-computer interaction in production and life scenes, there is a lack of a safe collaboration strategy that can enable humans and collaborative robots to understand each other. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a human-robot bidirectional cognition safe interaction method, which not only meets the safety design guidelines of robot behavior and the like when performing safety interaction such as "handover task" between humans and robots, but also dynamically adjusts the robot behavior through bidirectional intention cognition, thereby building a safe interaction space while ensuring efficiency.

[0009] To achieve the above-mentioned purpose of the application, the human-robot bidirectional cognition safe interaction method comprises the following steps:

[0010] (1) Training a perception model

[0011] 1.1) Physiological signal acquisition

[0012] In a hand task handover task, a physiological signal acquisition device is used to acquire skin electric response signals (GSR signals) and photoplethysmography signals (PPG signals) of multiple test personnel when interacting with the robot in real time;

[0013] 1.2) Data preprocessing

[0014] First, segmenting: the acquired GSR signals and PPG signals are cut into n GSR signal samples and n PPG signal samples according to a window size of w seconds and a window moving step of seconds;

[0015] Then, safety or unsafe label marking: n GSR signal samples and n PPG signal samples are labeled according to the safety or unsafe feelings reported by the test personnel, with safety being 1 and unsafe being 0;

[0016] Then, in the n GSR signal samples and the n PPG signal samples, randomly select a safe sample, constituting a safe sample pair and put into the training set, randomly select the same time window an unsafe sample, constituting an unsafe sample pair and put into the training set;

[0017] Finally, filtering: for the selected for the safe sample pair and for each GSR signal sample in the safe sample pair and

[0018] 1.3), build a perception model and train

[0019] Build a perception model including a cross-attention module and a deep neural network, and train:

[0020] 1.3.1), select a safe sample pair or an unsafe sample from the training set and send it into the cross-attention module, take the GSR signal sample as the query item, the PPG signal sample as the key item and the value item, calculate the cross-attention weight to calculate the GSR representation fused with PPG information, take the PPG signal sample as the query item, the GSR signal sample as the key item and the value item, calculate the cross-attention weight to calculate the PPG representation fused with GSR information;

[0021] 1.3.2), combine the GSR representation and the PPG representation into a feature matrix and send it into the deep neural network for feature extraction and prediction to obtain the prediction result, update the deep neural network parameters according to the loss of the prediction result and the sample label;

[0022] Repeat steps 1.3.1), 1.3.2) until the set number of iterations is reached or the classification result and the sample label loss is less than the set threshold, complete the training of the perception model;

[0023] (2), the robot perceives the safety of the human

[0024] 2.1), average safety index calculation

[0025] First, for the user, obtain the GSR signal and PPG signal according to step 1.1), then cut according to the window size w seconds and the window moving step seconds, and apply a median smoothing filter to the cut GSR signal sample for preprocessing, and apply a third-order Butterworth filter to the cut PPG signal sample for preprocessing;

[0026] Then, the GSR signal sample and the PPG signal sample of the current moment are sent into the perception model trained in step (1) to obtain a prediction result, i.e., 1 or 0.

[0027] Finally, the prediction result of the current moment is averaged with the previous k prediction results to obtain an average safety index, so that an average safety index is output every second;

[0028] 2.2), the robot performs behavior adaptive adjustment according to the average safety index

[0029] When the average safety index is lower than 0.1, the interaction distance and the robot movement speed are adjusted, when the average safety index is lower than 0.6 but higher than or equal to 0.1, only the interaction distance is adjusted, and when the average safety index is higher than or equal to 0.6, no adjustment is made.

[0030] (3), the intention of the robot and the working environment are enhanced by using augmented reality technology

[0031] On the basis of voice prompt type, i.e., through the robot to issue a language prompt to guide the user to perform the handover action, the robot also describes the state and behavior of the robot, i.e., the robot intention visualization, through the following three augmented reality technologies:

[0032] Action completion display type: real-time display of the action completion of the robot mechanical arm, wherein, display 100% indicates that the robot mechanical arm has reached the handover point;

[0033] End posture preview type: display of the final handover posture of the robot mechanical arm, when the robot mechanical arm reaches the final handover posture, it indicates that it has reached the handover point;

[0034] Trajectory preview type: display of the movement trajectory of the mechanical arm from the starting position to the handover position. The completion of the trajectory marks that the mechanical arm has reached the handover point.

[0035] The purpose of the present application is achieved.

[0036] The present application is based on the framework of augmented reality and physiological signals, and proposes a physiological signal driven augmented reality human-robot bidirectional cognitive safety interaction strategy. On the one hand, the physiological signals of the human are collected in real time by the sensor, and a physiological data driven perception prediction model is constructed, which is used to perceive the average safety index of the human, and the robot adjusts the behavior according to the average safety index, so as to enhance the cognition of the robot to the human body state (perceived safety, etc.), on the other hand, three kinds of augmented reality technologies are used to enhance the cognition of the human to the robot intention and working environment. In this way, when the human and the robot perform the "handover task" and other safety interactions, not only the safety design guidelines of the robot behavior, but also the dynamic adjustment of the robot behavior through the bidirectional intention cognition can be realized, so as to build a safe interaction space while ensuring the efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flow chart of a specific embodiment of the human-robot bidirectional cognitive safety interaction method of the present application;

[0038] Figure 2 is a schematic diagram of the principle of a specific embodiment of the human-robot bidirectional cognitive safety interaction method of the present application, wherein (a) is an architecture diagram of the human-robot bidirectional cognitive safety interaction method, and (b) is a process example of the human-machine handover task;

[0039] Figure 3 is a schematic diagram of the perception safety modeling process in the human-machine handover task of the present application, wherein (a) is equipment wearing, (b) is data collection (handover task), (c) is data preprocessing, (d) is perception safety modeling and prediction, and (e) is continuous safety index, i.e. average safety index;

[0040] Figure 4 is a subjective and objective data evaluation diagram, wherein (a) is an objective behavior data statistical diagram, and (b) is a subjective evaluation data statistical diagram. DETAILED DESCRIPTION

[0041] The specific embodiments of the present application will be described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present application. It should be particularly noted that in the following description, when the detailed description of the known functions and designs may obscure the main content of the present application, these descriptions will be omitted here.

[0042] In the augmented reality (AR) environment, the safety of human-machine interaction is a key consideration. The existing technology usually adopts one-way information transmission, i.e. the robot shows its behavior intention to the user through AR, but lacks real-time monitoring and response to the user's perceived safety. The present application proposes a human-robot bidirectional cognitive safety interaction method based on physiological signals, which is used for the interaction between human and robot in augmented reality.

[0043] In the present embodiment, as shown in Figure 1 , 2 The present inventor and the robot bidirectional cognitive safe interaction method comprises the following steps:

[0044] Step S1: training a perception model

[0045] Step S1.1: physiological signal acquisition

[0046] In the hand task handover task, the physiological signal acquisition device is used to collect the galvanic skin response signal (GSR signal) and the photoplethysmogram signal (PPG signal) of the multiple test persons when they interact with the robot in real time.

[0047] In the present embodiment, the physiological signal acquisition device uses a Shimmer3+ wireless wearable sensor device, which can provide real-time, simple and effective physiological and kinematic data acquisition for various application fields. The collected signals include IMU (9DoF through an accelerometer, a gyroscope, a magnetometer and an altimeter), electrocardiogram, EMG, GSR and PPG, etc. The powerful, flexible and configurable design of Shimmer3+ and the wireless sensor technology provide high-quality data acquisition, ensuring the scientific reliability of the experimental data of scientific researchers.

[0048] In the present embodiment, the galvanic skin response signal (GSR signal) and the photoplethysmogram signal (PPG signal) of 17 test persons when they interact with the robot are collected in real time. These signals reflect the emotional state and stress level of the test persons (users) and are key indicators for evaluating the user's perception of safety.

[0049] Due to the sensitivity of the signals to motion artifacts, in the specific implementation process, the participants are guided to act naturally, but at the same time, they are prevented from waving their hands excessively.

[0050] Step S1.2: data preprocessing

[0051] The collected physiological signals, i.e. the GSR signal and the PPG signal, are raw physiological signals and cannot be directly used for modeling because they exhibit the following characteristics: 1) non-stationary behavior: the statistical properties of the signals change over time, making signal processing complex; 2) complex characteristics: the features extracted from the signals are very complex and a detailed assessment of them is insufficient; 3) motion artifacts: the signals can contain fast transient artifacts that need to be removed through post-processing tools that apply noise filters; 4) individual differences: the same stimulus can cause different degrees of GSR response in different people. Therefore, a series of data preprocessing is performed:

[0052] First, segmentation is performed: the acquired GSR signal and PPG signal are cut respectively according to the window size w seconds and the window moving step s seconds, to obtain n GSR signal samples and n PPG signal samples.

[0053] In this embodiment, the window size w = 5 seconds, the window moving step s = 0.25 seconds, and n = 8006 GSR signal samples and n = 8006 PPG signal samples are obtained by cutting.

[0054] Then, safe or unsafe label marking is performed: the n GSR signal samples and the n PPG signal samples are labeled according to the safe or unsafe feeling reported by the tested person, safe as 1 and unsafe as 0.

[0055] In order to solve the long tail effect caused by individual differences, the label of the signal sample used for training is adjusted from five, i.e. 1 represents unsafe and 5 represents safe, to two, i.e. 0 represents unsafe and 1 represents safe.

[0056] Then, in the n GSR signal samples and the n PPG signal samples, randomly select safe samples in the same time window, to form safe sample pairs and put them into the training set, and randomly select unsafe samples in the same time window, to form unsafe sample pairs and put them into the training set.

[0057] In this embodiment, 3355, i.e. 3355 safe sample pairs and 3355 unsafe sample pairs are put into the training set.

[0058] Finally, in order to reduce the influence of noise and improve the signal quality, filtering is needed: for the selected safe sample pairs and unsafe sample pairs, a median smoothing filter is applied to each GSR signal sample for pretreatment, and a third-order Butterworth filter is applied to each PPG signal sample for pretreatment.

[0059] These data preprocessing steps are to ensure that the information extracted from the physiological signals is accurate and reliable, so as to be able to effectively model the perceived safety level.

[0060] Step S1.3: Constructing a perception model and training

[0061] ​Supervised learning has been proven to be able to detect cognitive states quickly, accurately and robustly by learning individual physiological features. Therefore, the present application constructs a perception model including a cross-attention module and a deep neural network, and trains it.

[0062] The present application draws on the bidirectional cross-attention and self-modal attention (BCSA) mechanism, which can effectively solve the delay problem between multi-modal physiological signals and more effectively align different signals to complement each other. Cross-attention is used to align and fuse two modalities of PPG signals and GSR signals. The present application maps time series signals to the same dimensional feature space to preserve more information and achieve multi-modal fusion.

[0063] Specifically, PPG signal samples are used as query items (Q), GSR signal samples are used as key items (K) and value items (V), and cross-attention weights are calculated to obtain PPG representations fused with GSR information. Conversely, GSR signal samples are used as query items, and PPG signal samples are used as key items and value items, and cross-attention weights are calculated to obtain GSR representations fused with PPG information. The cross-attention mechanism is used to exchange information between each PPG feature and GSR feature, producing cross-modal, which helps the network gradually focus on important single-modal time periods and potential alignment relationships between the two modalities. The specific calculation process can be represented by the following formula:

[0064]

[0065] The formula represents the cross-attention between PPG signal samples and GSR signal samples, where the PPG signal samples are query items and the GSR signal samples are key items and value items. 、 and are learned parameter matrices, is the i-th attention weight, representing the degree of matching between the PPG signal sample and , represents the i-th value of the GSR signal sample. The query item, key item and value item are mapped to the same dimension dk through different parameter matrices. This formula can be used to calculate the PPG representation fused with GSR information. Conversely, if the GSR signal sample is used as the query item and the PPG signal sample is used as the key item and the value item, the GSR representation fused with PPG information can be calculated, which is:

[0066] Step S1.3.1: Calculate the GSR representation fused with PPG information and the PPG representation fused with GSR information

[0067] A pair of safe sample pairs or unsafe samples is selected from the training set and sent into the cross-attention module, the GSR signal sample is taken as the query item, the PPG signal sample is taken as the key item and the value item, the cross-attention weight is calculated to obtain the GSR representation fused with the PPG information, the PPG signal sample is taken as the query item, the GSR signal sample is taken as the key item and the value item, the cross-attention weight is calculated to obtain the PPG representation fused with the GSR information.

[0068] In the embodiment, since the acquisition device itself has a delay of 25 ms to 100 ms, only one layer of cross-attention mechanism is used to ensure that it is updated at least once per second. If multiple layers of cross-attention mechanism are used, it may cause the update speed to be too slow.

[0069] The application uses a deep neural network for feature extraction and result prediction, specifically:

[0070] Step S1.3.2: combining the GSR representation and the PPG representation into a feature matrix for training a deep neural network

[0071] The GSR representation and the PPG representation are combined into a feature matrix and sent into a deep neural network for feature extraction and prediction to obtain a prediction result, and the deep neural network parameters are updated according to the loss of the prediction result and the sample label.

[0072] In order to improve the feature quality and generalization ability, and avoid human bias, in the embodiment, a deep neural network with 5 convolutional layers, 1 pooling layer and 1 fully connected layer is used to automatically learn the features of the time series signal, instead of manually extracting the GSR signal features processed by wavelet transform. We choose the convolutional layer as the feature extraction method because it can capture the short-term local dependence pattern of PPG and GSR signals, which is a feature of time series signals.

[0073] Steps S1.3.1 and S1.3.2 are repeated until a set number of iterations is reached or the loss of the classification result and the sample label is less than a set threshold, and the training of the perception model is completed.

[0074] Step S2: the robot perceives the safety of the person

[0075] Step S2.1: average safety index calculation

[0076] As shown in Figure 3 , first, for a user, as shown in Figure 3 (a), the device is worn, the GSR signal and the PPG signal are obtained according to step S1.1, as shown in Figure 3 (c), and then the window size is w seconds and the window moving step is The GSR signal samples are cut off every second, and the cut-off GSR signal samples are preprocessed by a median smoothing filter. The PPG signal samples are cut off every second, and the cut-off PPG signal samples are preprocessed by a third-order Butterworth filter.

[0077] Then, the current GSR signal sample and the current PPG signal sample are input into the trained perception model in step S1 to obtain a prediction result, i.e., 1 or 0, as shown in (d) of FIG. 6. Figure 3

[0078] The discrete prediction result can be very sensitive to mutations and unstable fluctuations, because it can only capture the overall trend of the data and ignore small changes in the data. This can lead to misjudgment of the participant's perceived safety state. Alireza pointed out that the advantages of converting the discrete human state prediction result into a continuous value include not only being able to adapt to sudden changes in physiological signals caused by devices or environments, but also being able to more accurately reflect changes in the participant's emotional state level instead of categorizing it into a few discrete categories. Because human emotions are continuous, different people may have different reactions to different stimuli, and emotional levels will also change over time and in different contexts. Therefore, using a continuous indicator can better adapt to individual differences and dynamic changes, reduce misjudgment of the participant's perceived safety, and then better help the participant manage perceived safety.

[0079] Therefore, as shown in (e) of FIG. 6, finally, the prediction result Figure 3 at the current time t is averaged with the previous K prediction results to obtain an average safety index:

[0080] where t represents the time sequence number, and each time interval is

[0081] 1 second, so that an average safety index is output every 1 second.

[0082] In this embodiment, i.e., an average safety index is output every 1 second:

[0083]

[0084] The entire data collection (handover task) is shown in (b) of FIG. 6. Figure 3

[0085] Step S2.2: The robot performs behavior self-adaptive adjustment according to the average safety index

[0086] The average safety index is a measure calculated based on human physiological signals. It can reflect continuous perceived safety in human-machine interaction. By monitoring the average safety index in real time, the robot can adjust its behavior to adapt to the participant's perceived safety state.​​​ We can dynamically adjust the interaction distance and speed of the robot before the handover task according to the changes, to ensure the safety of human-robot interaction. Through further analysis of the active reports of the subjects, we obtain the threshold for adjusting the behavior and state of the robot. For "fast motion + close distance interaction", 16 out of 17 subjects actively reported a sense of insecurity. Among them, 62.5% of the subjects (monitoring safety index less than 0.1) require adjustment of the interaction distance and the speed of the robot motion. 37.5% of the subjects (monitoring safety index less than 0.6 but greater than or equal to 0.1) only require adjustment of the interaction distance. Thus, the behavior adaptive adjustment according to the average safety index of the present application is obtained:

[0087] When the average safety index is less than 0.1, adjust the interaction distance and the speed of the robot motion, when the average safety index is less than 0.6 but greater than or equal to 0.1, only adjust the interaction distance, and when the average safety index is greater than or equal to 0.6, do not adjust.

[0088] The following algorithm shows the process of the robot distance-speed adaptive adjustment strategy based on SI proposed by us. This adjustment is based on the following algorithm:

[0089]

[0090] Step S3: Enhance human cognition of robot intention and working environment by using augmented reality technology

[0091] On the basis of voice prompt type, that is, guiding the user to perform the handover action by issuing a language prompt by the robot, the robot also describes the state and behavior of the robot, that is, the visualization of the robot intention, by visualizing the following three augmented reality technologies:

[0092] Action completion display type: real-time display of the action completion of the robot mechanical arm, wherein a display of 100% indicates that the robot mechanical arm has reached the handover point;

[0093] End pose preview type: display of the final handover pose of the robot mechanical arm, which indicates that the robot mechanical arm has reached the handover point when it reaches the final handover pose;

[0094] Trajectory preview type: display of the motion trajectory of the mechanical arm from the starting position to the handover position. The completion of this trajectory marks that the mechanical arm has reached the handover point.

[0095] In addition to (a) the voice prompt type (baseline), this invention designs three AR visualization methods to describe the robot's state and behavior, helping users complete human-robot collaborative tasks (i.e., handover) in a perceptible and safe manner. These three AR-based robot intent visualization methods are: (b) action completion display type, (c) end pose preview type, and (d) trajectory preview type, to enhance the user's understanding of the interaction process. For the voice prompt type (baseline), when the robot's arm reaches the handover point, the robot will issue a voice prompt "Please reach out" to guide the user to perform the "reach out" action. After 3 seconds, the robot will issue a voice prompt "Please catch" to guide the user to perform the "catch" action.

[0096] Results Display:

[0097] This invention was validated through experiments with 26 participants. Compared to existing technologies, it demonstrated significant effectiveness in improving user-perceived safety, reducing the number of failed handover tasks, decreasing the number of premature attempts, and enhancing the acceptability and usability of interaction strategies. In the validation study, we used one-way ANOVA (using Welch's method) to compare behavioral measurements among the three groups. Statistical results are as follows: Figure 4 As shown:

[0098] An asterisk indicates statistical significance. The term "baseline" refers to Group A, which did not employ the two-way cognitive safe interaction strategy. "R" "H" represents Group B, which adopted a one-way cognitive safe interaction strategy (using only AR to visualize robot intentions). Finally, "R" "H" indicates Group C, which employs a two-way cognitive safety interaction method, namely this invention (using AR to visualize robot intentions and utilizing physiological signals to model human perception of safety). From Figure 4 It can be seen that the present invention slightly increases the interaction duration, but reduces objective behaviors such as the number of interaction failures and premature hand reaching. At the same time, subjective evaluation data such as perceived safety, acceptance of interaction strategies, and usability of strategies are improved, which meets the safety design guidelines for robot behavior, and builds a safe interaction space while ensuring efficiency.

[0099] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for safe two-way cognitive interaction between humans and robots, characterized by comprising the following steps: (1) Training the perceptual model; 1.1) Physiological signal acquisition; In the hand-hand task handover, the skin conductance response signal (GSR) and photoplethysmography (PPG) signals of multiple subjects interacting with the robot were collected in real time using physiological signal acquisition equipment. 1.2) Data preprocessing; First, segment the data: The acquired GSR and PPG signals are segmented into segments with a window size of w seconds and a window movement step of [value missing]. The signal is cut in seconds to obtain n GSR signal samples and n PPG signal samples; Then, safety or unsafety labels are applied: n GSR signal samples and n PPG signal samples are labeled according to the safety or unsafety feelings reported by the test subjects, with 1 for safety and 0 for unsafety. Then, from the n GSR signal samples and n PPG signal samples, the same time window is randomly selected respectively. Each labeled as safe constitutes a sample. The safe sample pairs are added to the training set, and samples from the same time window are randomly selected. Each labeled as an unsafe sample constitutes... Unsafe sample pairs are added to the training set; Finally, filtering is performed: for the selected... For safe sample pairs and For unsafe sample pairs, each GSR signal sample is preprocessed using a mean smoothing filter, and each PPG signal sample is preprocessed using a third-order Butterworth filter. 1.3) Construct and train the perception model; Construct a perception model that includes a cross-attention module and a deep neural network, and train it: 1.3.1) Select a safe or unsafe sample pair from the training set and send it to the cross-attention module. Use the GSR signal sample as the query term and the PPG signal sample as the key and value term. Calculate the GSR representation that incorporates the PPG information by calculating the cross-attention weights. 1.3.2) Combine the GSR representation and PPG representation into a feature matrix and feed it into a deep neural network for feature extraction and prediction. Obtain the prediction result and update the parameters of the deep neural network based on the loss of the prediction result and sample labels. Repeat steps 1.3.1 and 1.3.2 until the set number of iterations is reached or the loss between the prediction result and the sample label is less than the set threshold, thus completing the training of the perceptual model; (2) The robot recognizes human perception of safety; 2.1) Calculation of average safety index; First, for the user, obtain the GSR and PPG signals according to step 1.1), then, with a window size of w seconds and a window movement step size of... The signal is cut in seconds, and the cut GSR signal samples are preprocessed using a medium-range smoothing filter, while the cut PPG signal samples are preprocessed using a third-order Butterworth filter. Then, the GSR signal sample and PPG signal sample at the current moment are fed into the perception model trained in step (1) to obtain the prediction result, i.e., 1 or 0. Finally, the current prediction result is averaged with the previous k-fold prediction results to obtain the average safety index. This is done every... Output an average safety index every second; 2.2) The robot adaptively adjusts its behavior based on the average safety index; When the average safety index is below 0.1, adjust the interaction distance and robot movement speed; when the average safety index is below 0.6 but above or equal to 0.1, only adjust the interaction distance; when the average safety index is above or equal to 0.6, do not make any adjustments. (3) Utilize augmented reality technology to enhance human perception of robot intentions and the working environment; In addition to voice prompts, where the robot guides the user through handover actions by issuing verbal prompts, the robot also uses the following three augmented reality technologies to visualize its state and behavior, i.e., robot intent visualization: Action completion display type: Real-time display of the robot arm's action completion status, where 100% indicates that the robot arm has reached the handover point; End posture preview: Displays the final handover posture of the robot arm. When the robot arm reaches the final handover posture, it means that it has reached the handover point. Track preview: Displays the movement trajectory of the robotic arm from the starting position to the handover position. The completion of this trajectory indicates that the robotic arm has reached the handover point.

2. The human-robot bidirectional cognitive safe interaction method according to claim 1, characterized in that, A cross-attention module uses only one layer of cross-attention mechanism.

Images