An eye-tracking wheelchair robot control method based on convolutional neural network

Abstract

The present invention relates to the technical field of eye-tracking wheelchair robots, and in particular to a control method for an eye-tracking wheelchair robot based on a convolutional neural network. The method comprises: identifying an eye movement area image through a convolutional neural network model to obtain a user's preliminary control instruction; serializing the eye movement characteristics processed by the convolutional neural network model to form time series data; and inputting the time series data into a long short-term memory neural network to determine whether to execute the control instruction. The present invention integrates the convolutional neural network and the long short-term memory network in series to improve the accuracy and robustness of the eye movement data, and realizes a steering control method for an eye-tracking wheelchair robot, and a logic design for misoperation detection and diagnosis. When the user looks to the left or right, he may want to turn, or he may just observe the situation on the left and right sides. The long short-term memory network model is used to judge and feedback the user's thoughts in real time, thereby increasing the system's ability to correct misoperations of steering.

Description

Technical Field

[0001] The present invention relates to the technical field of eye-tracking wheelchair robots, and in particular to a control method of an eye-tracking wheelchair robot based on a convolutional neural network. Background Art

[0002] Eye tracking technology plays an important role in assistive devices and human-computer interaction. Traditional eye tracking systems are usually based on laser technology and identify the user's gaze point by capturing the movement of the eyeball on the retina. These systems have made significant progress in assistive communication, human-computer interaction, and assistance for people with disabilities. However, existing eye tracking systems have certain limitations in terms of mobility and real-time performance, especially when applied to fields such as wheelchair robot control, which require higher accuracy and fast response.

[0003] As one of the important technologies in the field of deep learning, convolutional neural network (CNN) has been widely used in image recognition, object detection, artificial intelligence and other fields. CNN has good feature extraction and classification capabilities, and can learn and recognize complex visual patterns from a large amount of image data. In the field of eye steering, CNN-based eye steering methods have made some progress, but their application in electric wheelchair control has not been fully explored.

[0004] In addition, the human eye moves left and right in a habit of quickly passing in front when looking from left to right. When a person wants to turn left, he will look to the left. As the direction changes, the eye will return to the center position at a certain speed. If the person looks to the left but does not want to turn left, the eye should return to the center position at a very fast speed. This may lead to a misoperation, i.e., a misjudgment of the turn. Summary of the invention

[0005] The present invention discloses a method for controlling an eye-tracking wheelchair robot based on a convolutional neural network. The specific method is as follows:

[0006] The image of the user's eye movement area is collected by an infrared high-definition camera;

[0007] The convolutional neural network model is used to identify the eye movement area image and obtain the user's preliminary control instructions;

[0008] The eye movement features processed by the convolutional neural network model are serialized to form time series data;

[0009] The time series data is input into the long short-term memory neural network to determine whether to execute the control instruction.

[0010] Furthermore, three horizontal ultrasonic sensors and two downwardly tilted ultrasonic sensors are installed in front of the wheelchair robot. The horizontal detection range of the three horizontal ultrasonic sensors is 1 meter, the covered area is 8 centimeters, and a detection angle of 43 degrees is provided; the two tilted ultrasonic sensors are located between the horizontal ultrasonic sensors;

[0011] The obstacle distance is obtained through the ultrasonic sensor, and when the obstacle distance is less than the threshold, the wheelchair robot is controlled to stop.

[0012] Furthermore, the convolutional neural network model is used to identify the eye movement area image and obtain the user's preliminary control instructions. The specific method is as follows:

[0013] Before feeding the eye movement area images to the convolutional neural network model, they were decimated to 64×64 images, the mean image correlation was centered at zero, and normalized to unit variance by dividing by the standard deviation. The specific formula is as follows:

[0014]

[0015] Among them, I is the original image, I N is the preprocessed image, I mean and σ I are the mean and standard deviation of the original image respectively;

[0016] When training a convolutional neural network model, the mean square error is an error function that quantifies the quality of the classifier by measuring the classifier's fitness for the data. The formula is as follows:

[0017]

[0018] Where E is the loss function, y i and t i are the predicted value and true value of the i-th category, N c Represents the number of classes

[0019] The CNN is trained using a stochastic gradient descent optimization algorithm with an adaptive learning rate. Backpropagation involves iteratively calculating the gradient of the loss function. The model parameter update formula is as follows:

[0020]

[0021]

[0022] in is the weight of the connection between the kth neuron in the lth layer and the ith neuron in the (l-1)th layer, is the bias of the kth neuron in the lth layer, and ε is the learning rate.

[0023] Furthermore, the user's preliminary control instructions include:

[0024] Right gaze, that is, turning right;

[0025] Look ahead, that is, go straight;

[0026] Left gaze, that is, turning left;

[0027] Close your eyes and stop the operation.

[0028] Furthermore, time series data is formed, and the specific method is as follows:

[0029] The convolutional neural network model is used to identify the eye direction in the eye movement area image and construct a feature vector;

[0030] Vectorize the features, i.e. flatten or reshape them into one-dimensional vectors;

[0031] Divide the feature vector into different time steps according to the preset time interval;

[0032] Using a time window of fixed length, continuous feature vectors are taken as the data of a time window to form a time series.

[0033] Furthermore, the input of the LSTM neural network is a three-dimensional tensor, where the first dimension represents the number of time steps, the second dimension represents the dimension of the feature vector at each time step, and the third dimension represents the batch size;

[0034] The dimensions of the feature vector at each time step include eyes facing left, eyes facing right, and looking straight ahead;

[0035] The outputs of the LSTM neural network include:

[0036] Confirm left turn, cancel left turn, confirm right turn and confirm straight ahead.

[0037] Furthermore, the specific method of training the long short-term memory neural network is as follows:

[0038] Initialize the long short-term memory neural network model parameters;

[0039] When the wheelchair robot's motion matches the user's expectations, the user does not provide any feedback;

[0040] When the wheelchair robot movement does not match the user's expectations, the user rotates his eyes 360 degrees;

[0041] Record the historical data generated during user usage as training data, and use the training data to train the long short-term memory neural network.

[0042] Furthermore, the training data includes: the time that the eyes continue to face left or the time that the eyes continue to face right or the duration of direct gaze, and whether it is correctly labeled.

[0043] Furthermore, the wheelchair robot also includes a main control unit, a motor drive module, a wheelchair robot control module and a power management unit;

[0044] The main control unit controls the wheelchair robot control module through the motor drive module, and the power management unit provides working power for the wheelchair robot control module.

[0045] Due to the adoption of the above scheme, the present invention has the following beneficial effects:

[0046] 1. The present invention integrates convolutional neural networks and long short-term memory networks in series to improve the accuracy and robustness of eye movement data, and realizes an eye movement wheelchair robot steering control method, which can provide more convenient and efficient mobility solutions for disabled people suffering from multiple sclerosis, locked-in syndrome, muscular dystrophy, arthritis, Parkinson's disease, spinal cord injury and other diseases.

[0047] 2. The present invention provides a logic design for misoperation detection and diagnosis. When the user looks to the left or right, he may want to turn, or he may just be observing the situation on the left or right. The long short-term memory network model is used to judge and provide real-time feedback on the user's thoughts, thereby increasing the system's ability to correct misoperations of steering.

[0048] 3. The present invention designs an auxiliary command system, which uses a series of ultrasonic sensors to interface with the main control unit to stop the wheelchair robot in an emergency, helping the wheelchair to pass through crowded places safely.

[0049] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings of the present invention are as follows.

[0051] Figure 1 This is a schematic diagram of the electrical control structure of the wheelchair robot.

[0052] Figure 2 Schematic diagram of the convolutional neural network model workflow.

[0053] Figure 3 Schematic diagram of the workflow of the long short-term memory network model.

[0054] Figure 4 It is a schematic diagram of the overall workflow of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0056] The electronic control structure of the eye tracking wheelchair robot is as follows: Figure 1 As shown, three horizontal ultrasonic sensors and two downwardly inclined ultrasonic sensors are also installed in front of the wheelchair robot.

[0057] The ultrasonic safety device needs to cover all the range of activities of the wheelchair robot. An aluminum frame is installed on the wheelchair to fix the sensors. The horizontal detection range of the three outward sensors is 1 meter, 1 meter away from the aluminum frame, and the interval between the covered areas is 8 cm. In addition, the three sensors provide a detection angle of 43 degrees. Two sensors are placed between the first three sensors, but tilted downward. These sensors can detect the distance information between the wheelchair and the surrounding obstacles. The parking distance is 1m from the detected object, and the wheelchair robot stops instantly.

[0058] The wheelchair robot control system utilizes a main control unit to process the intelligent delay trigger strategy and the transmission data of the ultrasonic safety device, and controls the wheelchair robot through a motor drive module with the processed result; the original controller remains unchanged to maintain the original manual operation to control the wheelchair to move forward, turn left and turn right, and the joystick is modified externally. The Arduino Uno microcontroller is connected to the main control unit through a USB interface. The main control unit receives outputs from a misoperation detection and diagnosis system and an auxiliary command system, and controls two servo motors accordingly to realize the forward, stop and left and right directions of the controller joystick. The movement control of the wheelchair robot in any direction is realized through a combination of different rotation angles of the two servo motors, and the power management unit directly controls the main power switch of the wheelchair robot.

[0059] An eye-tracking wheelchair robot control method based on convolutional neural network, such as Figure 4 As shown, the specific steps are as follows:

[0060] S1. Collect images of the user's eye movement area through an infrared high-definition camera and determine the distance to obstacles through an ultrasonic radar.

[0061] S2. When there are no obstacles within the preset range, the convolutional neural network model is used to identify the eye movement area image and obtain the user's preliminary control instructions.

[0062] In step S2, the user's preliminary control instructions include: gaze right, that is, turn right; look forward, that is, go straight; gaze left, that is, turn left; close eyes, that is, stop operation.

[0063] The convolutional neural network model recognizes the image process as follows Figure 2As shown, before feeding the eye movement area images to the convolutional neural network model, they are extracted to 64×64 images, the mean image is correlated, centered at zero, and normalized to unit variance by dividing by the standard deviation. The specific formula is as follows:

[0064]

[0065] Among them, I is the original image, I N is the preprocessed image, I mean and σ I are the mean and standard deviation of the original image respectively;

[0066] When training a convolutional neural network model, the mean square error is an error function that quantifies the quality of the classifier by measuring the classifier's fitness for the data. The formula is as follows:

[0067]

[0068] Where E is the loss function, y i and t i are the predicted value and true value of the i-th category, N c Represents the number of classes

[0069] The CNN is trained using a stochastic gradient descent optimization algorithm with an adaptive learning rate. Backpropagation involves iteratively calculating the gradient of the loss function. The model parameter update formula is as follows:

[0070]

[0071]

[0072] in is the weight of the connection between the kth neuron in the lth layer and the ith neuron in the (l-1)th layer, is the bias of the kth neuron in the lth layer, and ε is the learning rate.

[0073] Convolutional neural network (CNN) is the main network model to extract features. It uses convolutional layers to learn raw data, i.e., the pixel intensity of the eye image. It uses pooling layers to reduce the dimension of the activation map. It uses fully connected layers to equip the network with classification capabilities. The one with the highest classification accuracy is used as the final prediction result.

[0074] S3. Serialize the eye movement features processed by the convolutional neural network model to form time series data.

[0075] In step S3, time series data is formed, and the specific method is as follows:

[0076] S31, identifying the eye direction in the eye movement area image through a convolutional neural network model, and constructing a feature vector;

[0077] S32, vectorize the features, i.e., flatten or reshape them into one-dimensional vectors;

[0078] S33, dividing the feature vector into different time steps according to a preset time interval;

[0079] S34. Use a time window of fixed length and take continuous feature vectors as data of a time window to form a time series.

[0080] S4. Input the time series data into the long short-term memory neural network to determine whether to execute the control instruction.

[0081] In step S4, the input of the long short-term memory neural network is a three-dimensional tensor, the first dimension represents the number of time steps, the second dimension represents the feature vector dimension of each time step, and the third dimension represents the batch size; the feature vector dimension of each time step includes eyes facing left, eyes facing right, and looking straight; the output of the long short-term memory neural network includes: confirm left turn, cancel left turn, confirm right turn, and confirm straight go.

[0082] like Figure 3 As shown, the specific method of training long short-term memory neural network is as follows:

[0083] Initialize the long short-term memory neural network model parameters;

[0084] When the wheelchair robot's motion matches the user's expectations, the user does not provide any feedback;

[0085] When the wheelchair robot movement does not match the user's expectations, the user rotates his eyes 360 degrees;

[0086] Record the historical data generated during user usage as training data, and use the training data to train the long short-term memory neural network.

[0087] The training data includes: the time the eyes continue to face left, the time the eyes continue to face right, or the duration of direct gaze, and whether it is correctly labeled.

[0088] The long short-term memory network is used to serialize the eye movement data after CNN processing to form time series data for the input of the LSTM network. Then, an appropriate LSTM network structure is designed and constructed, including input layer, hidden layer, output layer, etc., so that the serialized eye movement data can be used to train the LSTM network model to learn the user's eye movement patterns and behavioral patterns.

[0089] If the LSTM network predicts that the user's eyes continue to look left / right, it may mean that the user wants to turn left / right. If the eyes move quickly from left to right, it may mean that the user is just observing the surrounding environment and does not need a delayed operation. If the eyes do not move to the left or right, a forward command is output.

[0090] According to the real-time prediction results and feedback information of the LSTM network, the delay trigger time and conditions are dynamically adjusted to adapt to the eye movement habits and behavioral characteristics of different users. When the movement of the wheelchair is consistent with the user's expectations, the user does not provide any feedback. When the movement of the wheelchair does not meet the user's expectations, the user will turn his eyes 360 degrees. After receiving the eye movement information, the system knows that this is wrong training data and adjusts the model accordingly.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A convolutional neural network-based eye-tracking wheelchair robot control method, characterized in that: The specific method is as follows: The image of the user's eye movement area is collected by an infrared high-definition camera; The convolutional neural network model is used to identify the eye movement area image and obtain the user's preliminary control instructions; The eye movement features processed by the convolutional neural network model are serialized to form time series data; Input the time series data into the long short-term memory neural network to determine whether to execute the control instruction; The specific method of forming time series data is as follows: The convolutional neural network model is used to identify the eye direction in the eye movement area image and construct a feature vector; Vectorize the features, i.e. flatten or reshape them into one-dimensional vectors; Divide the feature vector into different time steps according to the preset time interval; Use a fixed-length time window to treat continuous feature vectors as data in a time window to form a time series; The input of the LSTM neural network is a three-dimensional tensor, where the first dimension represents the number of time steps, the second dimension represents the dimension of the feature vector for each time step, and the third dimension represents the batch size; The dimensions of the feature vector at each time step include eyes facing left, eyes facing right, and looking straight ahead; The outputs of the LSTM neural network include: Confirm left turn, confirm right turn and confirm straight ahead; The specific method of training long short-term memory neural network is as follows: Initialize the long short-term memory neural network model parameters; When the wheelchair robot's motion matches the user's expectations, the user does not provide any feedback; When the wheelchair robot movement does not match the user's expectations, the user rotates his eyes 360 degrees; Record the historical data generated during user use as training data, and use the training data to train the long short-term memory neural network; The training data includes: the time the eyes continue to face left, the time the eyes continue to face right, or the duration of direct gaze, and whether it is correctly labeled.

2. The eye tracking wheelchair robot control method based on convolutional neural network as claimed in claim 1, characterized in that: The wheelchair robot is also equipped with three horizontal ultrasonic sensors and two tilted ultrasonic sensors tilted downward. The horizontal detection range of the three horizontal ultrasonic sensors is 1 meter, the covered area is 8 centimeters, and a detection angle of 43 degrees is provided; the two tilted ultrasonic sensors are located between the horizontal ultrasonic sensors; The obstacle distance is obtained through the ultrasonic sensor, and when the obstacle distance is less than the threshold, the wheelchair robot is controlled to stop.

3. The eye tracking wheelchair robot control method based on convolutional neural network as claimed in claim 1, characterized in that: The convolutional neural network model is used to identify the eye movement area image and obtain the user's preliminary control instructions. The specific method is as follows: Before feeding the eye movement area images to the convolutional neural network model, they were decimated to 64×64 images, the mean image correlation was centered at zero, and normalized to unit variance by dividing by the standard deviation. The specific formula is as follows: Among them, I is the original image, I N is the preprocessed image, I mean and σ I are the mean and standard deviation of the original image respectively; When training a convolutional neural network model, the mean square error is an error function that quantifies the quality of the classifier by measuring the classifier's fitness for the data. The formula is as follows: Where E is the loss function, y i and t i are the predicted value and true value of the i-th category, N c Represents the number of classes The CNN is trained using a stochastic gradient descent optimization algorithm with an adaptive learning rate. Backpropagation involves iteratively calculating the gradient of the loss function. The model parameter update formula is as follows: in is the weight of the connection between the kth neuron in the lth layer and the ith neuron in the (l-1)th layer, is the bias of the kth neuron in the lth layer, and ε is the learning rate.

4. The eye tracking wheelchair robot control method based on convolutional neural network as claimed in claim 1, characterized in that: User preliminary control instructions include: Right gaze, that is, turning right; Look ahead, that is, go straight; Left gaze, that is, turning left; Close your eyes and stop the operation.

5. The eye tracking wheelchair robot control method based on convolutional neural network as claimed in claim 1, characterized in that: The wheelchair robot also includes a main control unit, a motor drive module, a wheelchair robot control module and a power management unit; The main control unit controls the wheelchair robot control module through the motor drive module, and the power management unit provides working power for the wheelchair robot control module.

Images