Robot automatic driving control method and device based on eye movement data set

Abstract

The invention relates to the technical field of robot automatic driving control, in particular to a robot automatic driving control method and device based on an eye movement dataset, and the method comprises the steps: obtaining the eye movement dataset of a target driver in multiple traffic scenes; inputting the eye movement data set into a pre-constructed driver attention prediction model to obtain attention information of the target driver in multiple traffic scenes; and inputting the attention information into a pre-constructed human attention guidance automatic driving model to output a control instruction that the target robot is in an automatic driving state, and controlling the target robot to execute the control instruction. Therefore, the problems that in the related technology, the calculation cost is high, understanding of key complex scenes is not enough, and the safety of automatic driving of the robot is low are solved.

Description

Technical Field

[0001] The present application relates to the technical field of robot autonomous driving control, and in particular to a robot autonomous driving control method and device based on an eye movement data set. Background Art

[0002] Due to the complexity of the actual operating environment, open roads and various types of people who do not obey traffic rules, the environment faced by autonomous driving robots is uncertain and highly dynamic. How to avoid visual perception obstacles, solve high-precision map problems, correctly solve problems such as traffic light misrecognition, how to quickly identify important visual clues and understand risks in a crowded traffic operation scene, etc., these factors will affect the safety and popularity of robot autonomous driving, so the improvement of target detection algorithm performance is crucial in the application of autonomous driving robots.

[0003] In the related technology, the corresponding navigation information can be determined according to the target location information and the current location information of the current robot, and the obstacle information detected by the collision detector corresponding to each driving path can be obtained, and then the unmanned current robot can be controlled to avoid the corresponding obstacles and drive to the target location according to the navigation information and obstacle information; the characteristic parameters of the shifting mechanical arm, throttle mechanical leg, brake mechanical leg and clutch mechanical leg of the car driving robot can also be input into the fuzzy neural network to obtain the vehicle speed tracking control of the car driving robot.

[0004] However, related technologies have high computing costs and insufficient understanding of key complex scenarios, resulting in low safety of robot autonomous driving, which urgently needs improvement. Summary of the invention

[0005] The present application provides a robot autonomous driving control method and device based on an eye movement data set to solve the problems in the related technology, such as high computing cost and insufficient understanding of key complex scenes, resulting in low safety of robot autonomous driving.

[0006] The first aspect of the present application provides a robot autonomous driving control method based on an eye movement dataset, comprising the following steps: obtaining an eye movement dataset of a target driver in multiple traffic scenarios; inputting the eye movement dataset into a pre-built driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios; inputting the attention information into a pre-built human attention-guided autonomous driving model to output control instructions for the target robot to be in an autonomous driving state, and controlling the target robot to execute the control instructions.

[0007] Optionally, in one embodiment of the present application, obtaining an eye movement dataset of a target driver in multiple traffic scenarios includes: based on a pre-built driving simulation system, obtaining scene video data and eye movement video data of the target driver in the multiple traffic scenarios; based on the scene video data and the eye movement video data, obtaining pupil center information and line of sight estimation information of the target driver in the multiple traffic scenarios; and generating the eye movement dataset based on the pupil center information and the line of sight estimation information.

[0008] Optionally, in one embodiment of the present application, before inputting the eye movement dataset into a pre-built driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios, it also includes: constructing a visual feature extraction module of the driver attention prediction model by a pre-trained target dilated fully convolutional neural network; constructing a visual feature processing module of the driver attention prediction model by a target fully convolutional neural network including at least three convolutional layers; constructing a timing processing module of the driver attention prediction model by a target convolutional long short-term memory network; constructing the driver attention prediction model by the visual feature extraction module, the visual feature processing module, the timing processing module and eye movement weighted sampling.

[0009] Optionally, in one embodiment of the present application, the attention information is input into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, including: based on the global low-resolution edge-central visual information corresponding to the attention information, an edge visual encoding module in the pre-built human attention-guided autonomous driving model obtains a first visual feature of the target robot; based on the local high-resolution central data information corresponding to the attention information, a central visual encoding module in the pre-built human attention-guided autonomous driving model obtains a second visual feature of the target robot; according to the attention prediction module in the pre-built human attention-guided autonomous driving model, the machine attention information of the target robot is obtained; based on the first visual feature, the second visual feature and the machine attention information, the peripheral central vision combined encoding module in the pre-built human attention-guided autonomous driving model generates the control instruction.

[0010] Optionally, in one embodiment of the present application, after the attention information is input into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, it also includes: obtaining an evaluation condition for the control instruction; determining whether the control instruction meets the evaluation condition; if the control instruction does not meet the evaluation condition, regenerating the control instruction based on the pre-built human attention-guided autonomous driving model until the control instruction meets the evaluation condition.

[0011] The second aspect of the present application provides a robot automatic driving control device based on an eye movement data set, including: a first acquisition module, used to obtain an eye movement data set of a target driver in multiple traffic scenarios; a first input module, used to input the eye movement data set into a pre-built driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios; a second input module, used to input the attention information into a pre-built human attention-guided automatic driving model to output a control instruction for the target robot to be in an automatic driving state, and control the target robot to execute the control instruction.

[0012] Optionally, in one embodiment of the present application, the first acquisition module includes: a first acquisition unit, used to acquire scene video data and eye movement video data of the target driver in the multi-traffic scenario based on a pre-built driving simulation system; a second acquisition unit, used to acquire pupil center information and line of sight estimation information of the target driver in the multi-traffic scenario based on the scene video data and the eye movement video data; a first generation unit, used to generate the eye movement dataset based on the pupil center information and the line of sight estimation information.

[0013] Optionally, in one embodiment of the present application, it also includes: a first construction module, which is used to construct a visual feature extraction module of the driver attention prediction model by a pre-trained target dilated fully convolutional neural network before inputting the eye movement dataset into a pre-built driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios; a second construction module, which is used to construct a visual feature processing module of the driver attention prediction model by a target fully convolutional neural network including at least three convolutional layers; a third construction module, which is used to construct a timing processing module of the driver attention prediction model by a target convolutional long short-term memory network; and a fourth construction module, which is used to construct the driver attention prediction model by the visual feature extraction module, the visual feature processing module, the timing processing module and eye movement weighted sampling.

[0014] Optionally, in one embodiment of the present application, the second input module includes: a second generation unit, used to obtain the first visual feature of the target robot based on the global low-resolution edge-central visual information corresponding to the attention information, by the edge visual coding module in the pre-built human attention-guided autonomous driving model; a third generation unit, used to obtain the second visual feature of the target robot based on the local high-resolution central data information corresponding to the attention information, by the central visual coding module in the pre-built human attention-guided autonomous driving model; a fourth generation unit, used to obtain the machine attention information of the target robot according to the attention prediction module in the pre-built human attention-guided autonomous driving model; a fifth generation unit, used to generate the control instruction based on the first visual feature, the second visual feature and the machine attention information, by the peripheral central vision combined coding module in the pre-built human attention-guided autonomous driving model.

[0015] Optionally, in one embodiment of the present application, it also includes: a second acquisition module, used to input the attention information into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, and obtain an evaluation condition of the control instruction; a judgment module, used to judge whether the control instruction meets the evaluation condition; and a generation module, used to regenerate the control instruction based on the pre-built human attention-guided autonomous driving model when the control instruction does not meet the evaluation condition, until the control instruction meets the evaluation condition.

[0016] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot autonomous driving control method based on an eye movement data set as described in the above embodiment.

[0017] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned robot autonomous driving control method based on eye movement data set.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned robot autonomous driving control method based on eye movement data set.

[0019] The embodiment of the present application can input the target driver's eye movement data set in multiple traffic scenarios into a pre-built driver attention prediction model, thereby obtaining the target driver's attention information in multiple traffic scenarios, and input the attention information into a pre-built human attention-guided autonomous driving model to obtain the control instructions of the target robot in the autonomous driving state. Through the precise collection of the eye movement data set, the target driver's attention information is learned and converted into the basis for the control of the target robot, and the key goals are quickly grasped, the accuracy and robustness of the robot's autonomous driving model prediction are improved, and more help is provided for machine intelligent driving in complex scenarios. In this way, the problems of high computational cost and insufficient understanding of key complex scenarios in related technologies, resulting in low safety of robot autonomous driving, are solved.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a robot automatic driving control method based on an eye movement data set provided according to an embodiment of the present application;

[0023] Figure 2 A schematic block diagram of a pre-built driver attention prediction model provided according to an embodiment of the present application;

[0024] Figure 3 A schematic block diagram of a pre-built visual multi-resolution autonomous driving model under human attention guidance provided according to an embodiment of the present application;

[0025] Figure 4 A flowchart of the working principle of a robot automatic driving control method based on an eye movement dataset provided according to an embodiment of the present application;

[0026] Figure 5 A block diagram of a robot automatic driving control device based on an eye movement data set provided according to an embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes the robot automatic driving control method and device based on the eye movement data set of the embodiment of the present application with reference to the accompanying drawings. In view of the high computational cost mentioned in the above background technology and the insufficient understanding of key complex scenes, which leads to the low safety of robot automatic driving, the present application provides a robot automatic driving control method based on an eye movement data set, in which the eye movement data set of the target driver in multiple traffic scenes can be input into a pre-built driver attention prediction model, thereby obtaining the attention information of the target driver in multiple traffic scenes, and the attention information is input into a pre-built human attention-guided automatic driving model, and the control instructions of the target robot in the automatic driving state are obtained. Through the accurate collection of the eye movement data set, the attention information of the target driver is learned and converted into the basis for the control of the target robot, the key target is quickly grasped, the accuracy and robustness of the prediction of the robot automatic driving model are improved, and more help is provided for the intelligent driving of machines in complex scenes. Thus, the problems of high computational cost and insufficient understanding of key complex scenes in the related technology, which leads to low safety of robot automatic driving, are solved.

[0030] Specifically, Figure 1 This is a flowchart of a robot autonomous driving control method based on an eye movement dataset provided according to an embodiment of the present application.

[0031] like Figure 1 As shown, the robot automatic driving control method based on the eye movement dataset includes the following steps:

[0032] In step S101, an eye movement dataset of a target driver in multiple traffic scenarios is obtained.

[0033] It can be understood that the visual and other multi-sensor data in the embodiment of the present application not only covers the conventional situations of lane-following and car-tracking, but also expands various traffic operation scenarios such as intersections and cyclists encroaching on motor vehicle lanes, and generalizes the traffic operation scenarios and classifies the data, thereby constructing multiple traffic scenarios.

[0034] In some embodiments, the embodiments of the present application can utilize a pre-built driving simulation system to obtain the target driver's eye movement behaviors such as gaze and scanning during driving in multiple traffic scenarios, and collect, record and accurately calculate relevant eye movement video data in real time, thereby obtaining the target driver's eye movement data set in multiple traffic scenarios.

[0035] Optionally, in one embodiment of the present application, an eye movement dataset of a target driver in multiple traffic scenarios is obtained, including: based on a pre-built driving simulation system, obtaining scene video data and eye movement video data of the target driver in multiple traffic scenarios; based on the scene video data and the eye movement video data, obtaining pupil center information and line of sight estimation information of the target driver in multiple traffic scenarios; and generating an eye movement dataset based on the pupil center information and the line of sight estimation information.

[0036] In some embodiments, the driving simulation system pre-built in the embodiments of the present application may include a display screen, a driving kit, an eye tracker, etc., and the present application does not make any specific limitations.

[0037] Among them, the display screen can play the scene video data of the target driver during the simulated driving process in real time (such as actual data collected by multiple sensors such as RGBD cameras / public driving scene video data sets, etc., which are not specifically limited in this application).

[0038] The driving kit is equipped with a steering wheel to be close to real driving, a car-like seat, and a pedal system for acceleration and deceleration, and records the target driver's acceleration, deceleration, steering and other operational behaviors in real time during the simulated driving.

[0039] Eye tracking may include, but is not limited to, a dedicated eye tracking device and an eye movement analysis software that records and monitors the driver's eye movements, etc. This application does not impose any specific limitations.

[0040] Further, the embodiment of the present application takes into account the subsequent data verification of actual scene applications, and the eye tracking device can use a head-mounted eye tracker, which can include but is not limited to a computing unit, an ET (eye camera) and a FT (scene camera), etc., and the present application does not make specific restrictions. Among them, in the embodiment of the present application, the computing unit collects and processes FT and ET images through FT and ET cameras, realizes accurate and real-time eye tracking, and transmits the processed FT images and eye movement video data to the back-end program through the network, and the back-end program is responsible for the analysis and processing of FT images and eye movement video data. Data accuracy is very important. The target driver participating in the simulated driving can be an experienced driver with more than 3 years of experience (specifically, it can be set by a technician in this field according to the actual situation, and this application does not make specific restrictions). Before the actual data collection, three-point calibration is performed, and a short practical test is performed after calibration to verify that the gaze tracking is accurate. After the verification is accurate, the driving simulation system pre-built in the embodiment of the present application and the eye tracker are synchronized in time, the simulated driving and recording are started, and the eye tracker starts to collect eye movement video data.

[0041] Furthermore, the embodiment of the present application synchronizes the scene video data and the eye movement video data through timestamps at a later stage, adopts processing methods such as multi-focus and false positive gaze filtering, and uses eye movement behavior analysis and semantic SLAM (Simultaneous Localization and Mapping) to accurately annotate key event videos, and establish an eye movement dataset under multiple traffic scenarios. Among them, the eye movement dataset of the embodiment of the present application not only includes urban driving data under various weather and lighting conditions, but also includes situations of crisis and rare traffic operation scenarios, which can further improve the accuracy of the robot's automatic driving model. In addition, the scene video data includes the scene video and its description, and provides a set of timestamped sensor measurements, such as the speed and route of the vehicle, and timestamped human annotations for the description and reasons of the vehicle's actions, which can be specifically set by technicians in this field according to actual conditions, and this application does not make specific restrictions.

[0042] In addition, it should be noted that the visual attention of the target driver in the embodiment of the present application enables the target driver to quickly identify and locate potential dangers or important visual cues, such as suddenly disappearing pedestrians, intrusion of nearby cyclists, or changes in traffic lights, etc., which are not specifically limited in the present application. Among them, the embodiment of the present application can use a large driver attention data set to perform end-to-end training through a neural network to estimate the attention of the target driver.

[0043] In some embodiments, after acquiring scene video data and eye movement video data, the embodiment of the present application realizes accurate eye tracking in an autonomous driving environment, which is the fundamental, and then can use high-precision target driver eye movement video data to effectively learn the robot autonomous driving model. Among them, accurate eye tracking mainly involves two stages: pupil center detection based on image features and line of sight estimation based on machine learning.

[0044] Among them, the content of pupil center detection based on image features in the embodiment of the present application may include: selecting the eye area of ​​the target driver, then detecting the corneal reflection point and pupil center of the target driver, and then obtaining the pupil center information of the target driver.

[0045] It should be noted that the detection of the pupil center is affected by the presence of the corneal reflection point in the embodiment of the present application. In order to eliminate the influence of the corneal reflection point on the calculation of the pupil center, the embodiment of the present application first performs threshold processing on each frame of eye movement data, then extracts the contour information of the pupil, and then removes the pupil contour information within the range of the spot, and finally uses the remaining contour to perform ellipse fitting, and the center of the ellipse is the pupil center. Furthermore, the embodiment of the present application uses the human eye model to extract the position of the eyeball center in the camera coordinate system from the human eye image sequence, and performs Kalman filtering and other processing on the position result to obtain higher accuracy and robustness.

[0046] The content of sight line estimation based on machine learning in the embodiment of the present application may include: due to some differences in the physiological structure of each target driver's eyes, most current eye tracking systems require the target driver to focus on the calibration points in the scene video data for calibration, so as to eliminate the inherent physiological deviation between the visual axis and the optical axis of the human eye and obtain the true gaze point position. In other words, the embodiment of the present application can use the human eye data set to establish a highly robust human eye model through machine learning methods for sight line estimation.

[0047] Furthermore, the embodiments of the present application can generate an eye movement dataset through pupil center information obtained by pupil center detection based on image features and line of sight estimation information obtained by line of sight estimation based on machine learning.

[0048] In addition, it should be noted that the eye tracking model of the embodiment of the present application provides three main functions: extracting perception primitives to manipulate the behavior of the vehicle; finding image areas that the model needs to process with high resolution (i.e. pedestrians, traffic lights, road cones, etc., which are not specifically limited in this application); and using central vision to enhance the capabilities of the peripheral vision system.

[0049] Optionally, in one embodiment of the present application, before inputting the eye movement dataset into a pre-built driver attention prediction model to obtain the attention information of the target driver in multiple traffic scenarios, it also includes: a visual feature extraction module that constructs a driver attention prediction model by a pre-trained target dilated fully convolutional neural network; a visual feature processing module that constructs a driver attention prediction model by a target fully convolutional neural network including at least three convolutional layers; a timing processing module that constructs a driver attention prediction model by a target convolutional long short-term memory network; and a driver attention prediction model is constructed by a visual feature extraction module, a visual feature processing module, a timing processing module and eye movement weighted sampling.

[0050] In some embodiments, the driver attention prediction model pre-built in the embodiment of the present application may include, but is not limited to, a visual feature extraction module, a visual feature processing module, and a timing processing module, etc., and the present application does not make specific restrictions. Figure 2 shown.

[0051] Among them, the visual feature extraction module can be a pre-trained target dilated fully convolutional neural network, whose weights are fixed during the training process. For example, the embodiment of the present application can use AlexNet pre-trained by ImageNet as the visual feature extraction module, using the features on the conv5 layer, and its input size can be set to 1024×576 pixels. The feature map of AlexNet is upsampled to 64×36 pixels and then sent to the visual feature processing module.

[0052] The visual feature processing module can be a target full convolutional neural network, which can be composed of three convolutional layers. In the embodiment of the present application, each convolutional layer can have a 1×1 convolution kernel, and there is a dropout layer after each convolutional layer to process the visual features from the visual feature extraction module, reducing the dimension of the visual features from 256 dimensions to 8 dimensions. The specific setting can be made by technicians in this field according to actual conditions, and this application does not make specific restrictions.

[0053] The timing processing module can be a target convolutional long short-term memory network, whose kernel size can be 3×3, and includes a Gaussian smoothing layer (where σ can be set to 1.5, which is not specifically limited in this application) and a softmax layer. In addition, in an embodiment of the present application, the timing processing module can sequentially receive the visual features of consecutive video frames from the visual feature processing module, predict the attention map for each new timing, and finally output the probability distribution on a 64×36 grid, predicting the possibility that each area of ​​the video frame is viewed by the target driver. Cross entropy is used as a loss function to measure the matching degree of the predicted probability distribution with the true value.

[0054] In addition, it should be noted that the embodiment of the present application also introduces eye movement weighted sampling in constructing the driver attention prediction model. Among them, in the embodiment of the present application, eye movement weighted sampling uses which frames in the target driver's eye data set are more critical driving moments. During the training of the driver attention prediction model, the frames can be weighted according to their importance level to improve the performance of the model. The trained model combined with eye movement weighted sampling is verified through test videos, which can prove that the driver attention prediction model pre-constructed in the embodiment of the present application can successfully notice objects that require the target driver's attention, such as the car in front, pedestrians who may enter the lane, etc., and this application does not make specific restrictions.

[0055] For example, the embodiment of the present application can select DKL (Kullback-Leibler divergence, KL divergence), CC (Pearson correlation coefficient, Pearson correlation coefficient), NSS (Normalized Scanpath Saliency, normalized scan path saliency) and AUC (Area Under Curve, ROC curve area) as attention prediction indicators, and calculate the average prediction error of these four indicators on the test set. The calculation results show that the prediction results of the embodiment of the present application are almost indistinguishable from the true value judged by humans.

[0056] In step S102, the eye movement dataset is input into a pre-built driver attention prediction model to obtain the attention information of the target driver in multiple traffic scenarios.

[0057] As a possible implementation method, the embodiment of the present application can use a pre-built driver attention prediction model to analyze the eye movement dataset to obtain the attention distribution information of the target driver in multiple traffic scenarios.

[0058] In step S103, the attention information is input into a pre-built human attention-guided autonomous driving model to output control instructions for the target robot to be in an autonomous driving state, and the target robot is controlled to execute the control instructions.

[0059] During the actual implementation process, the embodiments of the present application can utilize a pre-trained robot autonomous driving model, use the acquired attention information as input, generate control instructions for the target robot in the autonomous driving state, and control the target robot to drive according to these control instructions.

[0060] Optionally, in one embodiment of the present application, attention information is input into a pre-built human attention-guided autonomous driving model to output control instructions for the target robot to be in an autonomous driving state, including: based on the global low-resolution edge-central visual information corresponding to the attention information, an edge visual coding module in the pre-built human attention-guided autonomous driving model obtains a first visual feature of the target robot; based on the local high-resolution central data information corresponding to the attention information, a central visual coding module in the pre-built human attention-guided autonomous driving model obtains a second visual feature of the target robot; according to the attention prediction module in the pre-built human attention-guided autonomous driving model, the machine attention information of the target robot is obtained; based on the first visual feature, the second visual feature and the machine attention information, a peripheral central vision combined coding module in the pre-built human attention-guided autonomous driving model generates a control instruction.

[0061] In some embodiments, the human attention-guided autonomous driving model pre-built in the embodiments of the present application is as follows: Figure 3 As shown, the model may include but is not limited to: an edge visual encoding module, an attention prediction module, a central visual encoding module and a peripheral central visual combined encoding module.

[0062] Among them, the edge vision encoding module may include a peripheral vision encoder, which is not specifically limited in this application and can extract the first visual feature, such as advanced convolutional visual feature, through a convolutional neural network.

[0063] The attention prediction module can learn the attention behavior of the target driver in a supervised learning manner based on the image gaze pairs collected by the system from the target driver's side, and obtain the machine attention information of the target robot.

[0064] The central visual encoding module may include a central visual encoder that selects a central position, cuts a high-resolution central image block, and extracts a second visual feature from the high-resolution image block.

[0065] The peripheral central vision combined encoding module can combine the visual features of the edge vision encoding module and the central vision encoding module, and predict a primary control command, namely the speed of the vehicle.

[0066] It can be understood that the robot autonomous driving model in the embodiment of the present application uses imitation learning to train the network in autonomous driving, and guides the robot autonomous driving model through the target driver's attention information to predict driving commands from video frames, such as speed / steering, etc., and continuously improve its performance.

[0067] In addition, it should be noted that the robot autonomous driving model in the embodiment of the present application imitates the visual edge central multi-resolution driving model of the target driver's vision, first processes the low-resolution video frames, and predicts the gaze position of the target driver, and then extracts high-resolution input from the predicted gaze position, and finally combines the global low-resolution information and the local high-resolution information to predict the vehicle's speed control.

[0068] Among them, in the embodiment of the present application, the visual edge central multi-resolution driving model is better than the single-resolution model with the same computational amount, and the target driver's attention improves the accuracy of the robot's autonomous driving model, especially in critical scenarios involving pedestrians.

[0069] Furthermore, there are two challenges in designing a visual edge-central multi-resolution driving model in the embodiment of the present application: the first challenge is to effectively combine the global low-resolution visual edge center and the local high-resolution central vision, and dynamically scan the entire frame. In order to solve this problem, the embodiment of the present application combines the two visions by combining a fovea planner or two independent visual planners. The second challenge is that the selection of the central position is a non-differentiable process, and how to dynamically guide the central vision to the key position. It can be understood that in the embodiment of the present application, the data set has a large amount of target driver scene video data and eye movement video data, especially when an emergency occurs, the target driver will pay attention to the most critical area. Then the embodiment of the present application can integrate the attention information of the target driver into the robot's autonomous driving model, which can not only improve the average performance of the model, but also bring higher performance gains at critical moments. Solve the non-differentiable process by guiding the central vision to where the target driver will look.

[0070] Optionally, in one embodiment of the present application, after the attention information is input into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, it also includes: obtaining evaluation conditions for the control instructions; determining whether the control instructions meet the evaluation conditions; if the control instructions do not meet the evaluation conditions, regenerating the control instructions based on the pre-built human attention-guided autonomous driving model until the control instructions meet the evaluation conditions.

[0071] In some embodiments, the embodiments of the present application may use the mean absolute error, the root mean square error and the correlation coefficient as evaluation conditions for the control instructions, and when the control instructions do not meet the evaluation conditions, regenerate the control instructions through a pre-built human attention-guided autonomous driving model until the control instructions meet the evaluation conditions.

[0072] It can be understood that the embodiment of the present application uses the mean absolute error, root mean square error and correlation coefficient as evaluation conditions for control instructions. Compared with the control speed instructions predicted by the non-central robot autonomous driving model, the performance of the embodiment of the present application is improved by no less than 10%. Judging from the results of the test of the clips whose video reason annotations contain the words "pedestrian", "single person" or "multiple people", the improvement in prediction performance is more significant in critical situations involving pedestrians. This proves that incorporating the target driver's attention information into the robot autonomous driving model can accurately handle critical situations and improve the interpretability of the model results. The visual edge central simulation learning method adopted can be extended to other driving control command learning, such as steering for lane tracking tasks, etc., and this application does not make specific restrictions.

[0073] The working principle of the robot automatic driving control method based on eye movement data set proposed in the embodiment of the present application is introduced below in conjunction with a specific embodiment.

[0074] in, Figure 4 The present invention is a flowchart of the working principle of a robot autonomous driving control method based on an eye movement dataset provided according to one embodiment of the present application.

[0075] Step S401: construct scene video data of the target driver during simulated driving using actual data collected by multiple sensors in the vehicle and a public driving scene video dataset.

[0076] Step S402: acquiring eye movement video data of the target driver in multiple traffic scenarios based on the pre-built driving simulation system and scene video data.

[0077] Step S403: Based on the scene video data and the eye movement video data, the pupil center of the target driver is detected through an eye tracking model, and the line of sight of the target driver is estimated, thereby constructing an eye movement dataset of the target driver in multiple traffic scenes.

[0078] Step S404: Acquire the attention information of the target driver in multiple traffic scenarios based on the pre-built driver attention prediction model.

[0079] Step S405: Determine the control instructions for the target robot to be in the autonomous driving state based on the pre-built human attention-guided autonomous driving model.

[0080] According to the robot automatic driving control method based on the eye movement data set proposed in the embodiment of the present application, the eye movement data set of the target driver in multiple traffic scenes can be input into the pre-built driver attention prediction model, thereby obtaining the attention information of the target driver in multiple traffic scenes, and inputting the attention information into the pre-built human attention-guided automatic driving model to obtain the control instructions of the target robot in the automatic driving state. By accurately collecting the eye movement data set, the attention information of the target driver is learned and converted into the basis for the control of the target robot, the key goals are quickly grasped, the accuracy and robustness of the robot automatic driving model prediction are improved, and more help is provided for machine intelligent driving in complex scenes. Thus, the problems of high computational cost and insufficient understanding of key complex scenes in the related technology, resulting in low safety of robot automatic driving, are solved.

[0081] Secondly, the robot automatic driving control device based on the eye movement data set proposed in the embodiment of the present application is described with reference to the accompanying drawings.

[0082] Figure 5 A block diagram of a robot automatic driving control device based on an eye movement dataset provided according to an embodiment of the present application.

[0083] like Figure 5 As shown, the robot automatic driving control device 50 based on the eye movement data set includes: a first acquisition module 100, a first input module 200 and a second input module 300.

[0084] The first acquisition module 100 is used to acquire the eye movement dataset of the target driver in multiple traffic scenarios.

[0085] The first input module 200 is used to input the eye movement data set into a pre-built driver attention prediction model to obtain the attention information of the target driver in multiple traffic scenarios.

[0086] The second input module 300 is used to input the attention information into a pre-built human attention-guided autonomous driving model to output control instructions for the target robot to be in an autonomous driving state and control the target robot to execute the control instructions.

[0087] Optionally, in one embodiment of the present application, the first acquisition module 100 includes: a first acquisition unit, a second acquisition unit and a first generation unit.

[0088] Wherein, the first acquisition unit is used to acquire scene video data and eye movement video data of the target driver in multiple traffic scenarios based on a pre-built driving simulation system.

[0089] The second acquisition unit is used to acquire pupil center information and sight line estimation information of the target driver in multiple traffic scenes based on the scene video data and the eye movement video data.

[0090] The first generating unit is used to generate an eye movement data set based on pupil center information and sight line estimation information.

[0091] Optionally, in one embodiment of the present application, it further includes: a first building module, a second building module, a third building module and a fourth building module.

[0092] Among them, the first construction module is used to construct a visual feature extraction module of the driver attention prediction model by a pre-trained target dilated fully convolutional neural network before inputting the eye movement dataset into the pre-built driver attention prediction model to obtain the attention information of the target driver in multiple traffic scenarios.

[0093] The second building module is used to build a visual feature processing module of a driver attention prediction model by a target fully convolutional neural network including at least three convolutional layers.

[0094] The third building module is used to build a timing processing module of the driver attention prediction model by the target convolutional long short-term memory network.

[0095] The fourth construction module is used to construct a driver attention prediction model by using a visual feature extraction module, a visual feature processing module, a timing processing module and eye movement weighted sampling.

[0096] Optionally, in one embodiment of the present application, the second input module 300 includes: a second generation unit, a third generation unit, a fourth generation unit and a fifth generation unit.

[0097] Among them, the second generation unit is used to obtain the first visual feature of the target robot based on the global low-resolution edge-central visual information corresponding to the attention information, by the edge visual encoding module in the pre-built human attention-guided autonomous driving model.

[0098] The third generating unit is used to obtain the second visual feature of the target robot based on the local high-resolution central data information corresponding to the attention information by a central visual encoding module in a pre-built human attention-guided autonomous driving model.

[0099] The fourth generation unit is used to obtain the machine attention information of the target robot according to the attention prediction module in the pre-built human attention-guided autonomous driving model.

[0100] The fifth generation unit is used to generate control instructions based on the first visual feature, the second visual feature and the machine attention information by combining the peripheral central vision with the encoding module in the pre-built human attention-guided autonomous driving model.

[0101] Optionally, in an embodiment of the present application, it further includes: a second acquisition module, a judgment module and a generation module.

[0102] Among them, the second acquisition module is used to input the attention information into a pre-built human attention-guided autonomous driving model to output the control instructions of the target robot in the autonomous driving state and obtain the evaluation conditions of the control instructions.

[0103] The judgment module is used to judge whether the control instruction meets the evaluation conditions.

[0104] A generation module is used to regenerate control instructions based on a pre-built human attention-guided autonomous driving model when the control instructions do not meet the evaluation conditions, until the control instructions meet the evaluation conditions.

[0105] It should be noted that the aforementioned explanation of the embodiment of the robot automatic driving control method based on the eye movement data set is also applicable to the robot automatic driving control device based on the eye movement data set of this embodiment, and will not be repeated here.

[0106] According to the robot automatic driving control device based on the eye movement data set proposed in the embodiment of the present application, the eye movement data set of the target driver in multiple traffic scenes can be input into the pre-built driver attention prediction model, thereby obtaining the attention information of the target driver in multiple traffic scenes, and inputting the attention information into the pre-built human attention-guided automatic driving model to obtain the control instructions of the target robot in the automatic driving state. By accurately collecting the eye movement data set, the attention information of the target driver is learned and converted into the basis for the control of the target robot, the key goals are quickly grasped, the accuracy and robustness of the robot automatic driving model prediction are improved, and more help is provided for machine intelligent driving in complex scenes. Thus, the problems of high computational cost and insufficient understanding of key complex scenes in the related technology, resulting in low safety of robot automatic driving, are solved.

[0107] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0108] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0109] When the processor 602 executes the program, the robot automatic driving control method based on the eye movement data set provided in the above embodiment is implemented.

[0110] Furthermore, the electronic device further comprises:

[0111] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0112] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0113] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0114] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0115] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0116] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0117] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned robot autonomous driving control method based on an eye movement data set.

[0118] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned robot autonomous driving control method based on eye movement data set.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0123] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A robot automatic driving control method based on eye movement data set, characterized in that: The following steps are involved: Obtain the target driver's eye movement dataset in multiple traffic scenarios; Inputting the eye movement dataset into a pre-built driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios; The attention information is input into a pre-built human attention-guided autonomous driving model to output control instructions for the target robot to be in an autonomous driving state, and the target robot is controlled to execute the control instructions.

2. The method according to claim 1, characterized in that The step of obtaining the eye movement dataset of the target driver in multiple traffic scenarios includes: Based on a pre-built driving simulation system, acquiring scene video data and eye movement video data of the target driver in the multiple traffic scenes; Based on the scene video data and the eye movement video data, obtaining pupil center information and sight line estimation information of the target driver in the multi-traffic scene; The eye movement dataset is generated based on pupil center information and the gaze estimation information.

3. The method according to claim 1, characterized in that: Before inputting the eye movement dataset into a pre-built driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios, the method further includes: The visual feature extraction module of the driver attention prediction model is constructed by a pre-trained target dilated fully convolutional neural network; A visual feature processing module of the driver attention prediction model is constructed by a target fully convolutional neural network including at least three convolutional layers; A time series processing module of the driver attention prediction model is constructed by a target convolutional long short-term memory network; The driver attention prediction model is constructed by the visual feature extraction module, the visual feature processing module, the timing processing module and eye movement weighted sampling.

4. The method according to claim 1, characterized in that The step of inputting the attention information into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state includes: Based on the global low-resolution edge-central visual information corresponding to the attention information, an edge visual encoding module in the pre-built human attention-guided autonomous driving model obtains a first visual feature of the target robot; Based on the local high-resolution central data information corresponding to the attention information, a central visual encoding module in the pre-built human attention-guided autonomous driving model obtains a second visual feature of the target robot; Obtaining machine attention information of the target robot according to an attention prediction module in the pre-built human attention-guided autonomous driving model; Based on the first visual feature, the second visual feature and the machine attention information, the control instruction is generated by the peripheral central vision combined with the encoding module in the pre-built human attention-guided autonomous driving model.

5. The method according to claim 1, characterized in that After inputting the attention information into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, the method further includes: Obtaining an evaluation condition of the control instruction; Determining whether the control instruction satisfies the evaluation condition; If the control instruction does not meet the evaluation condition, the control instruction is regenerated based on the pre-built human attention-guided autonomous driving model until the control instruction meets the evaluation condition.

6. A robot automatic driving control device based on an eye movement dataset, characterized in that: include: The first acquisition module is used to acquire the eye movement dataset of the target driver in multiple traffic scenarios; A first input module, configured to input the eye movement dataset into a pre-built driver attention prediction model to obtain attention information of the target driver in the multiple traffic scenarios; The second input module is used to input the attention information into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, and control the target robot to execute the control instruction.

7. The device according to claim 6, characterized in that The acquisition module comprises: A first acquisition unit is used to acquire scene video data and eye movement video data of the target driver in the multi-traffic scene based on a pre-built driving simulation system; A second acquisition unit is used to acquire pupil center information and sight line estimation information of the target driver in the multi-traffic scene based on the scene video data and the eye movement video data; The first generating unit is used to generate the eye movement data set based on pupil center information and the sight line estimation information.

8. The device according to claim 6, characterized in that Also includes: A first construction module is used to construct a visual feature extraction module of the driver attention prediction model by a pre-trained target dilated fully convolutional neural network before inputting the eye movement dataset into a pre-constructed driver attention prediction model to obtain the attention information of the target driver in the multiple traffic scenarios; A second building module is used to build a visual feature processing module of the driver attention prediction model by a target fully convolutional neural network including at least three convolutional layers; A third construction module is used to construct a time series processing module of the driver attention prediction model by a target convolutional long short-term memory network; The fourth construction module is used to construct the driver attention prediction model by using the visual feature extraction module, the visual feature processing module, the timing processing module and eye movement weighted sampling.

9. The device according to claim 6, characterized in that The second input module comprises: a second generating unit, configured to obtain a first visual feature of the target robot by an edge visual encoding module in the pre-built human attention-guided autonomous driving model based on the global low-resolution edge-central visual information corresponding to the attention information; a third generating unit, configured to obtain, based on the local high-resolution central data information corresponding to the attention information, a second visual feature of the target robot by a central visual encoding module in the pre-built human attention-guided autonomous driving model; a fourth generating unit, configured to obtain machine attention information of the target robot according to an attention prediction module in the pre-built human attention-guided autonomous driving model; The fifth generating unit is used to generate the control instruction based on the first visual feature, the second visual feature and the machine attention information by the peripheral central vision combined with the encoding module in the pre-built human attention guided autonomous driving model.

10. The device according to claim 6, characterized in that Also includes: A second acquisition module is used to input the attention information into a pre-built human attention-guided autonomous driving model to output a control instruction for the target robot to be in an autonomous driving state, and obtain an evaluation condition for the control instruction; A judging module, used to judge whether the control instruction satisfies the evaluation condition; A generation module is used to regenerate the control instruction based on the pre-built human attention-guided autonomous driving model when the control instruction does not meet the evaluation condition, until the control instruction meets the evaluation condition.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot autonomous driving control method based on an eye movement data set as described in any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the robot autonomous driving control method based on an eye movement data set as described in any one of claims 1 to 5.

13. A computer program product, characterized in that It includes a computer program which, when executed, is used to implement the robot automatic driving control method based on an eye movement data set as described in any one of claims 1 to 5.

Images