A mechanical arm path planning method based on emotional brain-computer interface

Abstract

The application relates to the technical field of human-computer interaction, in particular to a mechanical arm path planning method based on an emotional brain-computer interface, which comprises the following steps: S1, a user watches a video collected by a camera, visual tracks of the user are recorded through an eye tracker, and electroencephalogram data of the user are synchronously collected; S2, emotions of the user are analyzed according to the electroencephalogram signals, and the electroencephalogram signals are classified into positive emotions, neutral emotions and negative emotions; S3, eye movement data are analyzed, and three-dimensional space coordinates of a user's gaze are located; S4, positive and negative emotions of the user are valued in a three-dimensional space environment according to the three-dimensional space coordinates of the user's gaze, and a total reward value graph is obtained; and S5, a path is planned according to a greedy algorithm and the total reward value graph, until a target position is reached, and path planning is completed. The application improves the efficiency of mechanical arm path planning, and the planned path is more in line with human expectations.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a mechanical arm path planning method based on an emotional brain-computer interface. BACKGROUND

[0002] A brain-computer interface (BCI) provides technical support for robots to understand human intentions. Among them, an affective brain-computer interface (aBCI) is an important branch in the field of emotional intelligence, aiming to achieve more natural and friendly human-computer interaction by recognizing human emotions. The electroencephalogram features used to distinguish emotions mainly include power spectral density (PSD) and differential entropy (DE). The neural activity of the brain under different emotional states will show different power spectral densities in different frequency bands. The frequency bands commonly used for emotion recognition include delta (1-4 Hz), theta (4-8 Hz), alpha (8-13 Hz), beta (13-30 Hz), and gamma (30-50 Hz). In addition, differential entropy can reflect the dynamic changes of electroencephalogram signals, and according to the asymmetry of the left and right brain hemispheres, features related to the positive and negative values of emotions are reflected.

[0003] In the past research on path planning, the environment often needs to be modeled in advance. However, this method has poor generalization and cannot adapt well to unmodeled scenarios, and the time and computational costs of modeling are high. In addition, some researches based on the interaction between agents and the environment for reinforcement learning set the state space, action space and appropriate reward function for the agent, and through continuous exploration, learn a strategy with the highest reward value. However, reinforcement learning usually requires a large amount of interaction data to train the model, which will lead to high training costs in the real world. SUMMARY

[0004] The present application provides a mechanical arm path planning method based on an emotional brain-computer interface to solve the technical problems mentioned in the background art.

[0005] To achieve the above purpose, the technical solution of the present application is as follows:

[0006] The present application provides a mechanical arm path planning method based on an emotional brain-computer interface, comprising the following steps:

[0007] S1, the user watches the video collected by the camera in real time, records the visual trajectory of the user through an eye tracker, and synchronously collects the electroencephalogram data of the user;

[0008] S2, analyzing the user's emotion according to the electroencephalogram signal, and classifying the electroencephalogram signal into positive emotion, neutral emotion and negative emotion;

[0009] S3, analyzing the eye movement data and locating the three-dimensional space coordinates of the user's gaze;

[0010] S4, assigning the positive and negative emotions of the user to the three-dimensional space environment according to the three-dimensional space coordinates of the user's gaze, and obtaining a total reward value map;

[0011] S5, planning the path according to the greedy algorithm and the total reward value map until the target position is reached, and completing the path planning.

[0012] Further, the S2 specifically includes the following steps:

[0013] S21, first, the original electroencephalogram signal is preprocessed to remove the interference signal and non-electroencephalogram noise in the original electroencephalogram signal, and the preprocessed electroencephalogram signal is segmented with a first set time as a step and a second set time as a window length;

[0014] S22, the preprocessed electroencephalogram data of each segment is extracted into multiple different frequency band electroencephalogram signals by a band-pass filter;

[0015] S23, the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal are calculated, and then the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal are input into a support vector machine as features to classify each segment of electroencephalogram signal into positive emotion, neutral emotion and negative emotion.

[0016] Further, the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal in S23 are calculated as follows:

[0017]

[0018] DE = log2(PSD)

[0019] Wherein, PSD represents the average power spectral density of each frequency band electroencephalogram signal, DE represents the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal, FFT(·) is fast Fourier transform, len(·) is signal length, and EEG represents electroencephalogram signal.

[0020] Further, the S3 specifically includes the following steps:

[0021] S31, analyze the eye movement data obtained by sampling, determine whether the user is in a fixation eye movement or a saccade eye movement, and store all sampling points in the fixation eye movement or the first sampling point in the saccade eye movement in a fixation buffer;

[0022] S32, construct a robot arm coordinate system with the robot arm base as the origin, the operation space as the positive direction of the X axis, the horizontal right direction as the positive direction of the Y axis, and the vertical upward direction as the positive direction of the Z axis;

[0023] S33, every first set time, take out the eye movement data synchronized with the electroencephalogram data for a second set time;

[0024] S34, if the number of sampling points in the fixation buffer in the eye movement data within the second set time is only one, the user's emotion corresponding to the electroencephalogram signal within the second set time is forcibly rejudged as a neutral emotion; if the number of sampling points in the fixation buffer in the eye movement data within the second set time is greater than one, the average values of the horizontal and vertical coordinates of all data in the fixation buffer in the camera coordinate system are solved respectively, and the depth value corresponding to the average coordinates is obtained according to the depth image of the binocular camera;

[0025] S35, through the calibration relationship between the camera coordinate system and the robot arm coordinate system, the average values of the horizontal and vertical coordinates in the camera coordinate system and the depth value are converted into three-dimensional space coordinates in the robot arm coordinate system, so that the three-dimensional space coordinates of the user's fixation within the second set time in the robot arm coordinate system are located;

[0026] S36, cycle S34 to S35 until all the eye movement data of the second set time taken out in S33 is processed according to the steps of S34 to S35.

[0027] Further, the S31 specifically includes the following steps:

[0028] S311, first, every time a sampling point is obtained, the oldest sampling point is removed from the eye movement data buffer, and the latest sampling point is added to the eye movement data buffer;

[0029] S312, calculate the distance between each sampling point and the previous sampling point in the eye movement data buffer point by point through the speed threshold filtering algorithm; if the distance between a certain sampling point and the previous sampling point meets the standard of fixation eye movement, it is considered that fixation eye movement has occurred, and the sampling point is added to a fixation buffer; if the distance between a certain sampling point and the previous sampling point is greater than the threshold value, it is considered that saccade eye movement has occurred, the fixation buffer is emptied, and the sampling point is taken as the first point of the fixation buffer.

[0030] Further, the first set time is 100-300 ms, and the second set time is 1000-4000 ms.

[0031] Further, the S4 specifically comprises the following steps:

[0032] S41, an active three-dimensional space matrix representing positive emotions with all values of 0 is established, and a negative three-dimensional space matrix representing negative emotions with all values of 0 is established; the active three-dimensional space matrix and the negative three-dimensional space matrix constitute a three-dimensional space environment;

[0033] S42, if a certain segment of brain electrical signals is judged to be positive emotions, then according to the gaze three-dimensional coordinates analyzed from the eye movement data corresponding to the segment of brain electrical signals, the value 1 is assigned at the gaze coordinates of the active three-dimensional space matrix; on the contrary, if a certain segment of brain electrical signals is judged to be negative emotions, then the value -1 is assigned at the gaze coordinates of the negative three-dimensional space matrix;

[0034] S43, taking all coordinates with non-0 values as the centers of the spheres, the positive and negative emotions are diffused to the entire three-dimensional space environment at the same diffusion attenuation rate, and the active three-dimensional space matrix and the negative three-dimensional space matrix are filled, so as to form an active reward value map and a negative reward value map;

[0035] S44, finally, the active reward value map and the negative reward value map are added to obtain a total reward value map.

[0036] Further, the S5 specifically comprises the following steps:

[0037] S51, taking the coordinates of the center of the mechanical arm in the mechanical arm coordinate system as the starting point of path planning;

[0038] S52, sequentially traversing the surrounding N path points, and according to the greedy algorithm and the total reward value map, taking the coordinate with the maximum reward value in the surrounding N path points as the next point in the path;

[0039] S53, repeating S52 until the target position is reached, and completing path planning.

[0040] The beneficial effects of the present application are:

[0041] The present application can help the robot to plan the path on the basis of understanding the environment and the user's expectation, and the planned path is more in line with the user's expectation, by decoding the brain electrical signals to identify the user's positive and negative attitudes, and by the eye tracker to assign the user's attitude as a reward to the three-dimensional environment space.

[0042] The present application has higher generalization than the traditional path planning by modeling, and does not need modeling, without modeling time cost, and reduces the calculation cost;

[0043] In addition, the present application does not require a large amount of interaction data to train the model, and there is no time cost of training the model compared with the traditional path specification by a large model. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A layout diagram of a mechanical arm coordinate system in the present application;

[0045] Figure 2 A positive reward value diagram in an embodiment of the present application;

[0046] Figure 3 A negative reward value diagram in an embodiment of the present application;

[0047] Figure 4 A total reward value diagram in an embodiment of the present application;

[0048] Figure 5 A path planning diagram obtained in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be implemented in many other different forms, and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0050] Reference Figure 1 The embodiment of the present application provides a mechanical arm path planning method based on an emotional brain-computer interface, comprising the following steps:

[0051] S1, a user watches a video collected by a camera in real time, records visual trajectories of the user through an eye tracker, and synchronously collects electroencephalogram data of the user;

[0052] S2, emotions of the user are analyzed according to the electroencephalogram signals, and the electroencephalogram signals are classified into positive emotions, neutral emotions and negative emotions;

[0053] S3, eye movement data collected by the eye tracker are analyzed and three-dimensional space coordinates of a user's gaze are located;

[0054] S4, positive and negative emotions of the user are valued in a three-dimensional space environment according to the three-dimensional space coordinates of the user's gaze, and a total reward value diagram is obtained;

[0055] S5, a path is planned according to a greedy algorithm and the total reward value diagram until a target position is reached, and path planning is completed.

[0056] In some embodiments, the S2 specifically comprises the following steps:

[0057] S21, first, the original electroencephalogram signal is preprocessed, so as to remove the interference signal and non-electroencephalogram noise in the original electroencephalogram signal, and the electroencephalogram signal after preprocessing is segmented with a first set time as a step and a second set time as a window length;

[0058] The first set time is 100-300 ms, and the second set time is 1000-4000 ms. Preferably, the first set time is 200 ms, and the second set time is 3000 ms.

[0059] Specifically, first, a 50 Hz notch filter is used to remove the power frequency interference signal in the original electroencephalogram signal, and then a 0.1-50 Hz band-pass filter is used to remove the non-electroencephalogram noise in the original electroencephalogram signal, to obtain the processed electroencephalogram signal, and the processed electroencephalogram signal is segmented with 3000 ms as a window length and 200 ms as a step.

[0060] S22, the electroencephalogram data after preprocessing is extracted as electroencephalogram signals of multiple different frequency bands by a band-pass filter;

[0061] Specifically, the electroencephalogram signal after preprocessing is extracted as signals of frequency bands delta (1-4 Hz), theta (4-8 Hz), alpha (8-13 Hz), beta (13-30 Hz) and gamma (30-50 Hz) by a band-pass filter.

[0062] S23, the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal are calculated, and then the average power spectral density of each frequency band and the differential entropy of each lead electroencephalogram signal are input into a support vector machine (Support Vector Machine, SVM) as features, so as to classify each segment of electroencephalogram signal into positive emotion, neutral emotion and negative emotion.

[0063] In some embodiments, the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal in S23 are calculated according to the following formula:

[0064]

[0065] DE = log2(PSD)

[0066] Wherein, PSD represents the average power spectral density of each frequency band electroencephalogram signal, DE represents the average power spectral density of each frequency band electroencephalogram signal and the differential entropy of each lead electroencephalogram signal, FFT(·) is fast Fourier transform, len(·) is signal length, and EEG represents electroencephalogram signal.

[0067] In some embodiments, the S3 specifically comprises the following steps:

[0068] S31, analyzing the sampled eye movement data to determine whether the user is in a fixation eye movement or a saccade eye movement, and storing all sampling points in the fixation eye movement or the first sampling point in the saccade eye movement into a fixation buffer;

[0069] S32, taking the base of the robot arm as the origin, pointing to the operation space as the positive direction of the X axis, horizontally to the right as the positive direction of the Y axis, and vertically upward as the positive direction of the Z axis, to construct a robot arm coordinate system;

[0070] S33, every 200 ms, taking out 3000 ms of eye movement data synchronized with the electroencephalogram data;

[0071] S34, if the number of sampling points in the fixation buffer in the 3000 ms eye movement data taken out is only 1, the user's emotion corresponding to the electroencephalogram signal synchronized with the 3000 ms eye movement data is forcibly changed to neutral emotion; if the number of sampling points in the fixation buffer in the 3000 ms eye movement data taken out is greater than 1, the average values of the horizontal and vertical coordinates of all data in the fixation buffer in the camera coordinate system are solved respectively, and the depth value corresponding to the average coordinates is obtained according to the depth image of the binocular camera;

[0072] S35, through the calibration relationship between the camera coordinate system and the robot arm coordinate system, the average values of the horizontal and vertical coordinates in the camera coordinate system and the depth value are converted into three-dimensional space coordinates in the robot arm coordinate system, so as to locate the three-dimensional space coordinates of the user's fixation in the robot arm coordinate system within 3000 ms;

[0073] S36, repeating S34 to S35 until all the second set time of eye movement data taken out in S33 is processed according to the steps of S34 to S35.

[0074] In some embodiments, the S31 specifically comprises the following steps:

[0075] S311, first, every time a sampling point is obtained, the oldest sampling point is removed from the eye movement data buffer, and the latest sampling point is added to the eye movement data buffer; the sampling frequency of the eye tracker is 60 Hz;

[0076] S312, calculate the distance between each sampling point and the last sampling point in the eye movement data buffer by the speed threshold filtering algorithm; if the distance between a sampling point and the last sampling point meets the standard of fixation eye movement, it is considered that fixation eye movement has occurred, and the sampling point is added to a fixation buffer; if the distance between a sampling point and the last sampling point is greater than the threshold, it is considered that saccade eye movement has occurred, the fixation buffer is emptied, and the sampling point is taken as the first point of the fixation buffer.

[0077] In some embodiments, the S4 specifically comprises the following steps:

[0078] S41, establish a positive three-dimensional space matrix representing positive emotions with all values being 0, and a negative three-dimensional space matrix representing negative emotions with all values being 0; the positive three-dimensional space matrix and the negative three-dimensional space matrix constitute a three-dimensional space environment;

[0079] S42, if a certain 3000ms electroencephalogram signal is judged to be positive emotions, the fixation three-dimensional coordinates analyzed from the eye movement data corresponding to the electroencephalogram signal are assigned a value of 1 at the fixation coordinates of the positive three-dimensional space matrix; otherwise, if a certain 3000ms electroencephalogram signal is judged to be negative emotions, a value of -1 is assigned at the fixation coordinates of the negative three-dimensional space matrix;

[0080] S43, take all coordinates with non-0 values as the centers of the spheres, and diffuse the positive and negative emotions to the entire three-dimensional space environment at the same diffusion decay rate, fill the positive three-dimensional space matrix and the negative three-dimensional space matrix, thereby forming a positive reward value map and a negative reward value map; the schematic diagrams of the positive reward value map and the negative reward value map are shown in Figure 2 and Figure 3 ;

[0081] The diffusion decay rate has the following characteristics:

[0082] The planned path avoids the points with a value of -1 in the negative reward value map and reaches the points with a value of 1 in the positive reward value map. By adjusting the diffusion decay rate, the planned path can be adjusted. The smaller the decay rate, the larger the arc of the obstacle avoidance; the larger the decay rate, the smaller the arc of the obstacle avoidance.

[0083] S44, finally add the positive reward value map and the negative reward value map to obtain a total reward value map. The schematic diagram of the total reward value map is shown in Figure 4 .

[0084] In some embodiments, the S5 specifically comprises the following steps:

[0085] S51, take the coordinates of the center of the mechanical hand in the mechanical arm coordinate system as the starting point of path planning;

[0086] S52, sequentially traverse the surrounding N path points, and according to the greedy algorithm and the total reward value map, the coordinate with the maximum reward value in the surrounding N path points is taken as the next point in the path; preferably, the value of N is 26;

[0087] Specifically, in the 3*3*3 cube centered on the current coordinate, the surrounding 26 path points are sequentially traversed, and according to the greedy algorithm and the total reward value map, the coordinate with the maximum reward value in the surrounding 26 path points is taken as the next point in the path;

[0088] S53, loop S52 until the target position is reached, and the path planning is completed. The completed path planning diagram is shown in Figure 5 .

[0089] The application recognizes the positive and negative attitudes of the user by decoding the electroencephalogram signal, and assigns the attitude of the user to the three-dimensional environment space as a reward through the eye tracker, which helps the robot to plan the path on the basis of understanding the environment and the user's expectations, and the planned path is more in line with the user's expectations. The efficiency of the robot arm path planning is improved.

[0090] The application has higher generalization than the traditional path planning through modeling, and does not need modeling, has no modeling time cost, and reduces the calculation cost;

[0091] In addition, the application does not need a large amount of interactive data to train the model, and has no time cost for training the model.

[0092] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Moreover, the technical solutions of each embodiment of the application can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method for emotional brain-computer interface based robot path planning, characterized in that, It comprises the following steps: S1, a user watches a video collected in real time, records visual tracks of the user through an eye tracker, and synchronously collects electroencephalogram data of the user; S2, emotions of the user are analyzed according to the electroencephalogram signals, and the electroencephalogram signals are classified into positive emotions, neutral emotions and negative emotions; S3, eye movement data are analyzed and three-dimensional space coordinates of a gaze of the user are located; S4, positive and negative emotions of the user are valued in a three-dimensional space environment according to the three-dimensional space coordinates of the gaze of the user, and a total reward value graph is obtained; S5, a path is planned according to a greedy algorithm and the total reward value graph until a target position is reached, and path planning is completed; The S2 specifically comprises the following steps: S21, first, original electroencephalogram signals are preprocessed, so that interference signals and non-electroencephalogram noise in the original electroencephalogram signals are removed, and the preprocessed electroencephalogram signals are segmented by a first set time as a step and a second set time as a window length; S22, each segment of the preprocessed electroencephalogram data is extracted into electroencephalogram signals of multiple different frequency bands through a band-pass filter; S23, average power spectral densities of the electroencephalogram signals of each frequency band and differential entropies of each lead electroencephalogram signal are calculated, and then the average power spectral densities of the electroencephalogram signals of each frequency band and the differential entropies of each lead electroencephalogram signal are input into a support vector machine as features to classify each segment of the electroencephalogram signals into positive emotions, neutral emotions and negative emotions; The S3 specifically comprises the following steps: S31, the sampled eye movement data are analyzed, it is determined that the user is in a gaze eye movement motion or a saccade eye movement motion, and all sampling points under the gaze eye movement motion or the first sampling point under the saccade eye movement motion is stored into a gaze buffer; S32, a mechanical arm base is taken as an origin, a pointing operation space is taken as a positive direction of an X axis, a horizontal right direction is taken as a positive direction of a Y axis, and a vertical upward direction is taken as a positive direction of a Z axis, and a mechanical arm coordinate system is constructed; S33, every first set time, eye movement data of a second set time synchronized with the electroencephalogram data are taken out; S34, if the number of the sampling points in the eye movement data in the second set time in the gaze buffer is only one, then the user emotion corresponding to the electroencephalogram signals in the second set time is forcibly changed to a neutral emotion; if the number of the sampling points in the eye movement data in the second set time in the gaze buffer is more than one, then average values of horizontal coordinates and vertical coordinates of all data in the gaze buffer in a camera coordinate system are respectively solved, and a depth value corresponding to the average coordinates is obtained according to a depth image of the binocular camera; S35, through a calibration relationship between the camera coordinate system and the mechanical arm coordinate system, the average values of the horizontal coordinates and the vertical coordinates and the depth value in the camera coordinate system are converted into three-dimensional space coordinates in the mechanical arm coordinate system, so that the three-dimensional space coordinates of the gaze of the user in the second set time in the mechanical arm coordinate system are located; S36, the steps from S34 to S35 are cycled until all the eye movement data of the second set time taken out in S33 are processed according to the steps from S34 to S35.

2. The method of claim 1, wherein, The average power spectral densities of the electroencephalogram signals of each frequency band and the differential entropies of each lead electroencephalogram signal in the S23 are calculated according to the following formulas: ； wherein, PSD represents the average power spectral density of the brain electrical signal of each frequency band, DE represents the average power spectral density of the brain electrical signal of each frequency band and the differential entropy of each lead brain electrical signal, FFT(·) is the fast Fourier transform, len(·) is the signal length, EEG represents the brain electrical signal.

3. The method of claim 2, wherein, The S31 specifically comprises the following steps: S311, first, every time a sampling point is obtained, the earliest sampling point is removed from the eye movement data buffer, and the latest sampling point is added to the eye movement data buffer; S312, the distance between each sampling point and the previous sampling point in the eye movement data buffer is calculated point by point through the speed threshold filtering algorithm; if the distance between a certain sampling point and the previous sampling point meets the standard of fixation eye movement, it is considered that fixation eye movement has occurred, and the sampling point is added to a fixation buffer; If the distance between a certain sampling point and the previous sampling point is greater than the threshold, it is considered that saccade eye movement has occurred, the fixation buffer is emptied, and the sampling point is taken as the first point of the fixation buffer.

4. The method of claim 1, wherein, The first set time is 100-300 ms, and the second set time is 1000-4000 ms.

5. The method of claim 1, wherein, The S4 specifically comprises the following steps: S41, an active three-dimensional space matrix representing positive emotions with all values being 0 is established, and a negative three-dimensional space matrix representing negative emotions with all values being 0 is established; The active three-dimensional space matrix and the negative three-dimensional space matrix constitute a three-dimensional space environment; S42, if a certain segment of brain electrical signals is judged to be positive emotions, the fixation three-dimensional coordinates analyzed from the eye movement data corresponding to the segment of brain electrical signals are assigned a value of 1 at the fixation coordinates of the active three-dimensional space matrix; On the contrary, if a certain segment of brain electrical signals is judged to be negative emotions, a value of -1 is assigned at the fixation coordinates of the negative three-dimensional space matrix; S43, taking all coordinates with non-zero values as the centers of the spheres, the positive and negative emotions are diffused to the entire three-dimensional space environment at the same diffusion decay rate, and the active three-dimensional space matrix and the negative three-dimensional space matrix are filled, thereby forming an active reward value map and a negative reward value map; S44, finally, the active reward value map and the negative reward value map are added to obtain a total reward value map.

6. The method of claim 1 to 5, wherein, The S5 specifically comprises the following steps: S51, taking the coordinates of the center of the mechanical hand in the mechanical arm coordinate system as the starting point of path planning; S52, the surrounding N path points are traversed in turn, and the coordinate with the maximum reward value in the surrounding N path points is taken as the next point in the path according to the greedy algorithm and the total reward value map; S53, loop S52 until the target position is reached, and the path planning is completed.

Images