Method and device for establishing eye movement data set based on multiple traffic operation scenes

Abstract

The invention relates to the technical field of vehicle driving, in particular to a method and device for establishing an eye movement data set based on multiple traffic operation scenes, and the method comprises the steps: generating the eye movement video data of a target driver in the multiple traffic operation scenes according to the scene video data of the multiple traffic operation scenes; acquiring a fixation hotspot map corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data; based on the fixation hotspot map, semantic segmentation information and three-dimensional reconstruction information of the fixation hotspot map are obtained through a pre-constructed LSD-SLAM, and an eye movement data set of the target driver is constructed based on the semantic segmentation information and the three-dimensional reconstruction information. Therefore, the problems that in the related technology, a data system of an eye movement combined operation scene lacks systematic data acquisition and analysis, data is poor, various traffic operation scenes are lacked, and the accuracy of marked data is low are solved.

Description

Technical Field

[0001] The present application relates to the field of vehicle driving technology, and in particular to a method and device for establishing an eye movement data set based on multiple traffic operation scenarios. Background Art

[0002] Human vision can provide key insights for autonomous driving robots to improve visual perception algorithms and enhance algorithm performance. Experienced drivers can quickly identify and locate dangerous or important visual signals that occur in blurred peripheral vision and make eye movements to these areas. By analyzing where drivers tend to focus their attention while driving, the sensor data processing of the autonomous driving robot can be adjusted, thereby significantly improving the performance of the autonomous driving robot model.

[0003] In the related technology, it is possible to construct a dangerous behavior recognition model by extracting vehicle behavior characteristics and eye movement speed from vehicle multi-dimensional data and driver's eye movement data, identify dangerous driving behavior, and generate a warning signal when dangerous driving behavior occurs; it is also possible to use an eye tracker to collect the driver's pupil diameter data, and use the pupil diameter during the traffic conflict period as a template for Fourier transform to generate template spectrum data, and then quickly identify traffic conflicts by setting a target threshold.

[0004] However, in the related technologies, the data system combining eye movement with operation scenarios lacks systematic data collection and analysis, resulting in poor data and a lack of diverse traffic operation scenarios. In addition, the accuracy of the labeled data is low, which urgently needs to be improved. Summary of the invention

[0005] The present application provides a method and device for establishing an eye movement data set based on multiple traffic operation scenarios to solve the problems in the related art that the data system combining eye movement with operation scenarios lacks systematic data collection and analysis, resulting in poor data, lack of diverse traffic operation scenarios, and low accuracy of labeled data.

[0006] The first aspect of the present application provides a method for establishing an eye movement dataset based on multiple traffic operation scenarios, comprising the following steps: generating eye movement video data of a target driver in the multiple traffic operation scenarios according to scene video data of the multiple traffic operation scenarios; obtaining a gaze heat map corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data; based on the gaze heat map, obtaining semantic segmentation information and three-dimensional reconstruction information of the gaze heat map by a pre-constructed LSD-SLAM (Large-Scale Direct SLAM, large-scale direct method simultaneous localization and mapping), so as to construct the eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information.

[0007] Optionally, in one embodiment of the present application, obtaining a gaze heat map corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data includes: extracting pupil center information of the target driver in each frame of eye movement data; obtaining line of sight estimation information of the target driver in each frame of eye movement data based on the pupil center information; and obtaining the gaze heat map based on the line of sight estimation information.

[0008] Optionally, in one embodiment of the present application, based on the gaze heat map, the semantic segmentation information and three-dimensional reconstruction information of the gaze heat map are obtained by a pre-constructed LSD-SLAM to construct the eye movement data set of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information, including: generating two-dimensional semantic segmentation information and two-dimensional semantic labels corresponding to each frame of eye movement data based on the SLAM in the pre-constructed LSD-SLAM; dividing the gaze heat map using the two-dimensional semantic labels to obtain a first gaze heat map of the gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map; based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map and the second semantic segmentation information, obtaining an optimized first gaze heat map corresponding to the first gaze heat map; based on the optimized first gaze heat map and the first semantic segmentation information, combining the recursive Bayesian and conditional random field in the pre-constructed LSD-SLAM to obtain the three-dimensional reconstruction information of the first gaze heat map; generating the eye movement data set based on the three-dimensional reconstruction information and the optimized first gaze heat map.

[0009] Optionally, in one embodiment of the present application, generating eye movement video data of a target driver in multiple traffic operation scenarios based on scene video data of the multiple traffic operation scenarios includes: determining key events in the multiple traffic operation scenarios; extracting scene key video frame data in the scene video data based on the key events; and determining the eye movement video data according to a scene timestamp corresponding to the scene key video frame data.

[0010] Optionally, in one embodiment of the present application, determining the eye movement video data according to the scene timestamp corresponding to the scene key video frame data includes: obtaining initial eye movement video data of the target driver in the multi-traffic operation scene according to the scene video data; and determining the eye movement video data based on the initial eye movement video data and the scene timestamp.

[0011] The second aspect of the present application provides a device for establishing an eye movement dataset based on multiple traffic operation scenarios, including: a generation module, used to generate eye movement video data of a target driver in the multiple traffic operation scenarios according to scene video data of the multiple traffic operation scenarios; an acquisition module, used to acquire a gaze heat map corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data; a construction module, used to obtain semantic segmentation information and three-dimensional reconstruction information of the gaze heat map by a pre-constructed LSD-SLAM based on the gaze heat map, so as to construct the eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information.

[0012] Optionally, in one embodiment of the present application, the acquisition module includes: a first extraction unit, used to extract the pupil center information of the target driver in each frame of eye movement data; an acquisition unit, used to obtain the line of sight estimation information of the target driver in each frame of eye movement data according to the pupil center information; and a first generation unit, used to obtain the gaze heat map based on the line of sight estimation information.

[0013] Optionally, in one embodiment of the present application, the construction module includes: a second generation unit, used to generate two-dimensional semantic segmentation information and two-dimensional semantic labels corresponding to each frame of eye movement data based on the SLAM in the pre-built LSD-SLAM; a division unit, used to divide the gaze heat map using the two-dimensional semantic labels to obtain a first gaze heat map of the gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map; a third generation unit, used to obtain an optimized first gaze heat map corresponding to the first gaze heat map based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map and the second semantic segmentation information; a fourth generation unit, used to obtain the three-dimensional reconstruction information of the first gaze heat map based on the optimized first gaze heat map and the first semantic segmentation information, combined with the recursive Bayes and conditional random field in the pre-built LSD-SLAM; a fifth generation unit, used to generate the eye movement dataset based on the three-dimensional reconstruction information and the optimized first gaze heat map.

[0014] Optionally, in one embodiment of the present application, the generation module includes: a first determination unit, used to determine the key events in the multi-traffic operation scene; a second extraction unit, used to extract scene key video frame data in the scene video data based on the key events; a second determination unit, used to determine the eye movement video data according to the scene timestamp corresponding to the scene key video frame data.

[0015] Optionally, in one embodiment of the present application, the second determination unit includes: an acquisition subunit, used to acquire the initial eye movement video data of the target driver in the multi-traffic operation scene according to the scene video data; and a determination subunit, used to determine the eye movement video data based on the initial eye movement video data and the scene timestamp.

[0016] An embodiment of the third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for establishing an eye movement data set based on multiple traffic operation scenarios as described in the above embodiment.

[0017] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above method of establishing an eye movement dataset based on multiple traffic operation scenarios.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above method of establishing an eye movement dataset based on multiple traffic operation scenarios.

[0019] The embodiment of the present application can generate eye movement video data based on scene video data of multiple traffic operation scenarios, and then use the gaze heat map corresponding to each frame of eye movement data to obtain semantic segmentation information and three-dimensional reconstruction information from the pre-constructed LSD-SLAM, so as to construct the target driver's eye movement data set. In addition to covering the conventional situations of robot lane-following and robot vehicle-tracking, it also expands multiple traffic operation scenarios such as intersections and cyclists encroaching on motor vehicle lanes, and performs traffic scene generalization and data classification. Through the smoothing of the target driver's eye movement video data, SLAM removes sky noise, identifies and annotates key events, and efficiently and accurately establishes the target driver's eye movement data set in multiple traffic operation scenarios. As a result, the problem of the lack of systematic data collection and analysis in the data system combining eye movement and operation scenarios in the related technology is solved, resulting in data poverty, lack of diverse traffic operation scenarios, and low accuracy of the annotated data.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1A flowchart of a method for establishing an eye movement dataset based on multiple traffic operation scenarios according to an embodiment of the present application;

[0023] Figure 2 A block diagram of a driving simulation system provided according to an embodiment of the present application;

[0024] Figure 3 A schematic diagram of an eye tracker provided according to an embodiment of the present application;

[0025] Figure 4 A schematic diagram of an example of a heat map provided according to an embodiment of the present application;

[0026] Figure 5 A schematic diagram of another example of a heat map provided according to an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a gaze trajectory diagram provided according to an embodiment of the present application;

[0028] Figure 7 A flowchart of constructing an eye movement data set of a target driver by a pre-built LSD-SLAM according to an embodiment of the present application;

[0029] Figure 8 A flowchart of semantic mapping provided according to an embodiment of the present application;

[0030] Fig. 9 A block diagram of an apparatus for establishing an eye movement dataset based on multiple traffic operation scenarios according to an embodiment of the present application;

[0031] Fig.10 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] The following describes the method and device for establishing an eye movement data set based on multiple traffic operation scenarios in the embodiment of the present application with reference to the accompanying drawings. In view of the lack of systematic data collection and analysis in the data system of eye movement combined with operation scenarios mentioned in the above background technology, which makes the data poor, lacks diverse traffic operation scenarios, and the accuracy of the labeled data is low, the present application provides a method for establishing an eye movement data set based on multiple traffic operation scenarios, in which eye movement video data can be generated according to the scene video data of multiple traffic operation scenarios, and then the gaze heat map corresponding to each frame of eye movement data is used to obtain semantic segmentation information and three-dimensional reconstruction information by the pre-constructed LSD-SLAM, so as to construct the eye movement data set of the target driver, in addition to covering the conventional situations of robot lane-following and robot vehicle-tracking, it also expands multiple traffic operation scenarios such as intersections and cyclists encroaching on motor vehicle lanes, and performs traffic scene generalization and data classification, through the target driver's eye movement video data smoothing, SLAM removes sky noise, and identifies and annotates key events, the target driver's eye movement data set under multiple traffic operation scenarios is efficiently and accurately established. This solves the problem in related technologies that the data system combining eye movement with operation scenarios lacks systematic data collection and analysis, resulting in poor data, lack of diverse traffic operation scenarios, and low accuracy of labeled data.

[0034] Specifically, Figure 1 The present invention provides a flowchart of a method for establishing an eye movement dataset based on multiple traffic operation scenarios according to an embodiment of the present application.

[0035] like Figure 1 As shown, the method for establishing an eye movement dataset based on multiple traffic operation scenarios includes the following steps:

[0036] In step S101 , eye movement video data of a target driver in a multi-traffic operation scene is generated according to scene video data of the multi-traffic operation scene.

[0037] It can be understood that in an embodiment of the present application, scene video data can be classified and labeled according to location and event, and multiple types of data such as scene video data, eye movement video data and various sensor data (including target driver operation behavior) of the same type can be aligned and aggregated together using a timestamp method.

[0038] Furthermore, in the embodiment of the present application, the eye movement video data may include, but is not limited to, three modes of video data including gaze data, saccade data, and following motion data of the target driver.

[0039] In addition, it should be noted that the target driver in the embodiment of the present application can be understood as a driver with more than 3 years of experience, which can be specifically set by technicians in this field according to actual conditions, and this application does not impose any specific restrictions.

[0040] As a possible implementation method, the embodiment of the present application can obtain eye movement video data generated by scene video data of the target driver in multiple traffic operation scenarios.

[0041] For example, the embodiments of the present application can be combined with Figure 2 The driving simulation system generates eye movement video data of the target driver.

[0042] Specifically, if Figure 2 As shown, the driving simulation system 20 includes: a display screen 201 , a driving kit 202 and an eye tracker 203 .

[0043] Among them, the display screen 201 can play scene video data of multiple traffic operation scenes (data actually collected by multiple sensors such as RGBD cameras / public driving video data sets).

[0044] The driving kit 202 is equipped with a steering wheel to be close to real driving, a car-like seat, and a pedal system for acceleration and deceleration, and records the target driver's acceleration, deceleration, steering and other operating behaviors in real time throughout the process.

[0045] The eye tracker 203 is composed of a dedicated eye tracking device and an eye movement analysis software that records and monitors the target driver's eye movements. Figure 3 shown.

[0046] Furthermore, the embodiments of the present application take into account the subsequent promotion of data verification for actual traffic scene applications, and the eye tracking device adopts a head-mounted eye tracker, which can be provided with but not limited to a computing unit, ET (eye camera) and FT (scene camera), etc., and the present application does not impose specific restrictions.

[0047] Among them, in the embodiment of the present application, the computing unit collects and processes FT and ET images through FT and ET cameras to achieve accurate and real-time eye tracking, and transmits the processed FT images and eye movement data to the back-end program through the network, and the back-end program is responsible for the analysis and processing of FT images and eye movement data. Data accuracy is very important, therefore, the embodiment of the present application performs three-point calibration before actual data collection, and performs a short practical test after calibration to verify that gaze tracking is accurate.

[0048] After the verification is accurate, the driving simulation system 20 and the eye tracker 203 are synchronized, the simulated driving and video recording are started, and the driving simulation system 20 starts to collect eye movement data. The scene video data and the eye movement video data are synchronized by time stamp later.

[0049] Eye movement analysis software can provide, but is not limited to, heat maps and gaze trajectory maps, etc. This application does not impose any specific restrictions.

[0050] The heat map can be understood as indicating where the target driver's eyes were looking for a few seconds, which is very useful for summarizing where the focus is directed. Blue shading indicates areas that were briefly or carefully looked at, and red shading indicates areas where the driver's eyes were focused for a longer period of time. Figure 4 and Figure 5 These are examples of heat maps provided in the embodiments of the present application, wherein: Figure 4 It describes a frame where the target driver’s gaze is mainly focused on the pedestrian part; Figure 5 It describes a frame that is smoothed by the aggregation of multiple target driver gazes, covering key traffic events.

[0051] The gaze trajectory diagram can be used to reveal the time sequence, location and time of observation of the target driver, which has an explanatory and guiding significance for event-based driving behavior. The trajectory points show the spatiotemporal relationship of the target driver's attention information in a broad sense (scattered points need to be filtered), such as Figure 6 The gaze trajectory diagram shown.

[0052] Optionally, in one embodiment of the present application, eye movement video data of a target driver in multiple traffic operation scenarios is generated based on scene video data of multiple traffic operation scenarios, including: determining key events in multiple traffic operation scenarios; extracting scene key video frame data in the scene video data based on the key events; and determining eye movement video data based on a scene timestamp corresponding to the scene key video frame data.

[0053] It can be understood that the key events in the embodiments of the present application can be understood as braking events, braking events, turning events, lane switching events and acceleration events, etc., and the present application does not make specific limitations.

[0054] In some embodiments, in order to effectively collect attention data in eye movement video data, the embodiments of the present application can first determine key events in multiple traffic operation scenarios, and then extract scene key video frame data based on the key events, and use the scene timestamp to determine the eye movement video data.

[0055] It should be noted that in the embodiment of the present application, the visual attention of the target driver enables the target driver to quickly identify and locate potential dangers or important visual cues, such as a suddenly disappeared pedestrian, an intrusion by a nearby cyclist, or a change in a traffic light. That is, the scene video data of the embodiment of the present application includes not only driving operation data under various weather and lighting conditions, but also situations of crisis and rare traffic scenes, such as various driving activities and environments including lane following, turning, switching lanes, and parking in chaotic scenes.

[0056] For example, in the embodiment of the present application, braking events, braking events, turning events, lane switching events, and acceleration events can be used as key events, and then video editing is performed 6.5 seconds before (or other values, which are not specifically limited in this application) and 3.5 seconds after (or other values, which are not specifically limited in this application) the key events, thereby obtaining more than 2,000 scene video data, which contain a large number of different road users and are detected using YOLO. It should be noted that each scene video data in the embodiment of the present application usually contains multiple vehicles and / or pedestrians, which are not specifically limited in this application.

[0057] Furthermore, the embodiment of the present application extracts scene key video frame data in each scene video data according to key events, and then obtains eye movement video data using the scene timestamp.

[0058] Optionally, in one embodiment of the present application, eye movement video data is determined according to a scene timestamp corresponding to scene key video frame data, including: obtaining initial eye movement video data of the target driver in a multi-traffic operation scenario according to the scene video data; and determining the eye movement video data based on the initial eye movement video data and the scene timestamp.

[0059] As a possible implementation method, the target driver in the embodiment of the present application can first obtain initial eye movement video data through scene video data in multiple traffic operation scenes, and then use the scene timestamp to determine the eye movement video data of the target driver.

[0060] It can be understood that each frame of the video scene in the scene video data of the embodiment of the present application can be regarded as an image sequence, which is much richer in content than an image, has strong expressiveness, and has a large amount of information. However, each frame of the video scene usually has a lot of redundancy, and the extraction of video frames also has the phenomenon of missing frames and redundancy. Therefore, the embodiment of the present application can extract the scene key video frame data in the scene video data through key events, so as to effectively reduce the time required for scene video data retrieval and enhance the accuracy of scene video data retrieval.

[0061] Furthermore, the embodiment of the present application finds the key frame of the initial eye movement video data according to the scene timestamp of the scene key video frame data, and then obtains the eye movement video data.

[0062] In step S102, a gaze heat map corresponding to each frame of eye movement data in the eye movement video data is obtained based on the eye movement video data.

[0063] From the above analysis, it can be seen that the embodiment of the present application can use eye movement analysis software to obtain the gaze heat map corresponding to each frame of eye movement data.

[0064] Optionally, in one embodiment of the present application, a gaze heat map corresponding to each frame of eye movement data in the eye movement video data is obtained based on the eye movement video data, including: extracting pupil center information of the target driver in each frame of eye movement data; obtaining line of sight estimation information of the target driver in each frame of eye movement data according to the pupil center information; and obtaining a gaze heat map based on the line of sight estimation information.

[0065] As a possible implementation method, the contents of the gaze heat map generated by the eye movement analysis software in the embodiment of the present application may include two parts: pupil center detection and line of sight estimation.

[0066] In some embodiments, the embodiments of the present application can detect the pupil center based on image features, and then obtain the pupil center information of the target driver in each frame of eye movement data.

[0067] Exemplarily, the embodiment of the present application first selects the eye area of ​​the target driver in each frame of eye movement data, then detects the corneal reflection point and pupil center of the target driver, and then obtains the pupil center information of the target driver.

[0068] It should be noted that the detection of the pupil center is affected by the presence of the corneal reflection point in the embodiment of the present application. In order to eliminate the influence of the corneal reflection point on the calculation of the pupil center, the embodiment of the present application first performs threshold processing on each frame of eye movement data, then extracts the contour information of the pupil, and then removes the pupil contour information within the range of the spot, and finally uses the remaining contour to perform ellipse fitting, and the center of the ellipse is the pupil center. Furthermore, the embodiment of the present application uses the human eye model to extract the position of the eyeball center in the camera coordinate system from the human eye image sequence, and performs Kalman filtering and other processing on the position result to obtain higher accuracy and robustness.

[0069] In some embodiments, the embodiments of the present application can perform line of sight estimation based on machine learning, and then obtain the line of sight estimation information of the target driver in each frame of eye movement data.

[0070] It can be understood that, in the embodiment of the present application, due to some differences in the physiological structure of each target driver's eyes, most current eye tracking systems require the target driver to focus on the calibration points in the scene video data for calibration, so as to eliminate the inherent physiological deviation between the human eye's visual axis and the optical axis and obtain the true gaze point position. In other words, the embodiment of the present application can use the human eye data set to establish a highly robust human eye model through machine learning methods for line of sight estimation.

[0071] In step S103, based on the gaze heat map, semantic segmentation information and three-dimensional reconstruction information of the gaze heat map are obtained by the pre-constructed LSD-SLAM, so as to construct an eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information.

[0072] It is understandable that the embodiments of the present application show through psychological research that when the target driver is faced with the need to pay attention to multiple visual cues at the same time, the order in which the target driver looks at these cues is very subjective, so the eye movement video data will encounter single focus and false positive gaze problems.

[0073] Furthermore, in the embodiment of the present application, a single focus can be understood as only recording one position that the target driver is observing at each moment, while the target driver may notice multiple important objects in the scene, and these objects are hidden, that is, a person's eyes are fixed on one object while paying attention to another object; a false positive gaze can be understood as the target driver moving his eyes to driving-irrelevant areas, such as the sky, trees, and buildings, thereby generating irrelevant eye movement video data.

[0074] Furthermore, in order to solve the above-mentioned problems, the embodiment of the present application requires more than 5 target drivers (or other values, which are not specifically limited in the present application) to participate in the collection of relevant data in the same scene video data. The heat map and gaze trajectory map formed by the eye movement video data of these target drivers are aggregated and smoothed to produce a gaze heat map for each frame of eye movement video data. The embodiment of the present application can record multiple important visual clues in one frame of video data by gathering the gaze of the target driver; it can remove irrelevant areas by averaging the eye movements of the target driver and combining the pre-built LSD-SLAM, thereby effectively removing noise sources such as the sky, trees and buildings; multi-focus and false positive gaze filtering can be achieved through eye movement aggregation smoothing and scene detection segmentation.

[0075] In the actual implementation process, the embodiment of the present application can obtain the semantic segmentation information and three-dimensional reconstruction information of the gaze heat map based on the gaze heat map by the pre-built LSD-SLAM, and then construct the eye movement dataset of the target driver.

[0076] Optionally, in one embodiment of the present application, based on the gaze heat map, semantic segmentation information and three-dimensional reconstruction information of the gaze heat map are obtained by a pre-built LSD-SLAM, so as to construct an eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information, including: generating two-dimensional semantic segmentation information and two-dimensional semantic labels corresponding to each frame of eye movement data based on the SLAM in the pre-built LSD-SLAM; dividing the gaze heat map using the two-dimensional semantic labels to obtain a first gaze heat map of the gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map; obtaining an optimized first gaze heat map corresponding to the first gaze heat map based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map, and the second semantic segmentation information; obtaining the three-dimensional reconstruction information of the first gaze heat map based on the optimized first gaze heat map and the first semantic segmentation information, combining the recursive Bayes and conditional random field in the pre-built LSD-SLAM; generating an eye movement dataset based on the three-dimensional reconstruction information and the optimized first gaze heat map.

[0077] Those skilled in the art can understand that the steps of constructing the target driver's eye movement data set by the pre-built LSD-SLAM in the embodiment of the present application are as follows: Figure 7 As shown, its main contents are:

[0078] Step S701: Generate two-dimensional semantic segmentation information and two-dimensional semantic labels for each frame of eye movement data using SLAM.

[0079] It is understandable that the embodiments of the present application can estimate the movement of the camera in SLAM, and the changes in the positions of various objects in each frame of eye movement data can also be predicted, generating a large amount of new data to provide more optimization conditions for semantic tasks and save the cost of manual calibration. Therefore, semantic mapping uses dense or semi-dense feature points, so that the two-dimensional semantic segmentation information and two-dimensional semantic labels (especially camera pose) obtained by SLAM can improve the performance of semantic segmentation. Among them, the flowchart of semantic mapping is as follows Figure 8 shown.

[0080] Step S702: Use two-dimensional semantic labels to divide the gaze heat map to obtain a first gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map.

[0081] Among them, the embodiment of the present application can map the two-dimensional semantic segmentation information of each frame of eye movement data, that is, pixels with two-dimensional semantic labels, into a three-dimensional point cloud, and then obtain a first gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map.

[0082] In addition, it should be noted that in the embodiment of the present application, the first gaze heat map can be understood as a gaze heat map that does not include targets such as the sky, trees, and buildings; the second gaze heat map can be understood as a gaze heat map that includes targets such as the sky, trees, and buildings.

[0083] Step S703: obtaining an optimized first gaze heat map based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map and the second semantic segmentation information.

[0084] Among them, the first gaze heat map of the embodiment of the present application performs semantic segmentation, and the second gaze heat map uses a comparison method based on small baseline stereo vision to optimize the depth estimation of the first gaze heat map, thereby obtaining the optimized first gaze heat map.

[0085] Step S704: Based on the optimized first gaze heat map and the first semantic segmentation information, three-dimensional reconstruction information is obtained using recursive Bayes and conditional random fields.

[0086] Among them, the embodiment of the present application can use recursive Bayesian enhanced semantic segmentation and use conditional random fields to optimize three-dimensional reconstruction information, so as to achieve semantic segmentation optimization and three-dimensional reconstruction.

[0087] It can be understood that the embodiment of the present application can use a recursive Bayesian method of semantic fusion. According to SLAM's estimation of pixel motion, the semantic classification probability of the pixel in the current frame is multiplied by the classification probability at the old position in the previous frame as the final probability, that is, the probability of the pixel will be multiplied along each frame, thereby enhancing the result of semantic segmentation.

[0088] Step S705: Generate an eye movement data set based on the three-dimensional reconstruction information and the optimized first gaze heat map.

[0089] According to the method for establishing an eye movement data set based on multiple traffic operation scenarios proposed in the embodiment of the present application, eye movement video data can be generated based on the scene video data of multiple traffic operation scenarios, and then the gaze heat map corresponding to each frame of eye movement data is used to obtain semantic segmentation information and three-dimensional reconstruction information from the pre-constructed LSD-SLAM, thereby constructing the eye movement data set of the target driver. In addition to covering the conventional situations of robot lane-following and robot vehicle-tracking, it also expands multiple traffic operation scenarios such as intersections and cyclists encroaching on motor vehicle lanes, and performs traffic scene generalization and data classification. Through the smoothing of the eye movement video data of the target driver, SLAM removes sky noise, identifies and annotates key events, and efficiently and accurately establishes the eye movement data set of the target driver in multiple traffic operation scenarios. Thus, the problem that the data system of eye movement combined with operation scenarios lacks systematic data collection and analysis in the related technology, resulting in data poverty, lack of diverse traffic operation scenarios, and low accuracy of annotated data is solved.

[0090] Next, a device for establishing an eye movement data set based on multiple traffic operation scenarios according to an embodiment of the present application is described with reference to the accompanying drawings.

[0091] Fig. 9 A block diagram of an apparatus for establishing an eye movement dataset based on multiple traffic operation scenarios according to an embodiment of the present application.

[0092] like Fig. 9 As shown, the device 90 for establishing an eye movement dataset based on multiple traffic operation scenarios includes: a generation module 100 , an acquisition module 200 and a construction module 300 .

[0093] The generating module 100 is used to generate the eye movement video data of the target driver in the multiple traffic operation scenes according to the scene video data of the multiple traffic operation scenes.

[0094] The acquisition module 200 is used to acquire the gaze hotspot corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data.

[0095] The construction module 300 is used to obtain semantic segmentation information and three-dimensional reconstruction information of the gaze heat map based on the gaze heat map by the pre-constructed LSD-SLAM, so as to construct the eye movement data set of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information.

[0096] Optionally, in one embodiment of the present application, the acquisition module 200 includes: a first extraction unit, an acquisition unit and a first generation unit.

[0097] Wherein, the first extraction unit is used to extract pupil center information of the target driver in each frame of eye movement data.

[0098] The acquisition unit is used to acquire the sight line estimation information of the target driver in each frame of eye movement data according to the pupil center information.

[0099] The first generating unit is used to obtain a gaze heat map based on the gaze estimation information.

[0100] Optionally, in one embodiment of the present application, the construction module 300 includes: a second generation unit, a division unit, a third generation unit, a fourth generation unit and a fifth generation unit.

[0101] Among them, the second generating unit is used to generate two-dimensional semantic segmentation information and two-dimensional semantic labels corresponding to each frame of eye movement data based on the SLAM in the pre-built LSD-SLAM.

[0102] A segmentation unit is used to segment the gaze heat map using two-dimensional semantic labels to obtain a first gaze heat map of the gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map.

[0103] The third generating unit is used to obtain an optimized first gaze heat map corresponding to the first gaze heat map based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map and the second semantic segmentation information.

[0104] The fourth generation unit is used to obtain the three-dimensional reconstruction information of the first gaze heat map based on the optimized first gaze heat map and the first semantic segmentation information in combination with the recursive Bayes and conditional random field in the pre-constructed LSD-SLAM.

[0105] The fifth generating unit is used to generate an eye movement data set based on the three-dimensional reconstruction information and the optimized first gaze heat map.

[0106] Optionally, in one embodiment of the present application, the generating module 100 includes: a first determining unit, a second extracting unit and a second determining unit.

[0107] Wherein, the first determination unit is used to determine key events in multiple traffic operation scenarios.

[0108] The second extraction unit is used to extract scene key video frame data from the scene video data based on the key event.

[0109] The second determining unit is used to determine the eye movement video data according to the scene timestamp corresponding to the scene key video frame data.

[0110] Optionally, in one embodiment of the present application, the second determining unit includes: an acquiring subunit and a determining subunit.

[0111] The acquisition subunit is used to acquire the initial eye movement video data of the target driver in multiple traffic operation scenarios according to the scene video data.

[0112] A determination subunit is used to determine the eye movement video data based on the initial eye movement video data and the scene timestamp.

[0113] It should be noted that the aforementioned explanation of the method embodiment for establishing an eye movement dataset based on multiple traffic operation scenarios is also applicable to the device for establishing an eye movement dataset based on multiple traffic operation scenarios of this embodiment, and will not be repeated here.

[0114] According to the device for establishing an eye movement data set based on multiple traffic operation scenarios proposed in the embodiment of the present application, eye movement video data can be generated based on the scene video data of multiple traffic operation scenarios, and then the gaze heat map corresponding to each frame of eye movement data is used to obtain semantic segmentation information and three-dimensional reconstruction information from the pre-constructed LSD-SLAM, thereby constructing the eye movement data set of the target driver. In addition to covering the conventional situations of robot lane-following and robot vehicle-tracking, it also expands multiple traffic operation scenarios such as intersections and cyclists encroaching on motor vehicle lanes, and performs traffic scene generalization and data classification. Through the smoothing of the eye movement video data of the target driver, SLAM removes sky noise, identifies and annotates key events, and efficiently and accurately establishes the eye movement data set of the target driver in multiple traffic operation scenarios. Thus, the problem that the data system of eye movement combined with operation scenarios lacks systematic data collection and analysis in the related technology, resulting in data poverty, lack of diverse traffic operation scenarios, and low accuracy of annotated data is solved.

[0115] Fig.10 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0116] A memory 1001 , a processor 1002 , and a computer program stored in the memory 1001 and executable on the processor 1002 .

[0117] When the processor 1002 executes the program, the method for establishing an eye movement dataset based on multiple traffic operation scenarios provided in the above embodiment is implemented.

[0118] Furthermore, the electronic device further comprises:

[0119] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .

[0120] The memory 1001 is used to store computer programs that can be executed on the processor 1002 .

[0121] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0122] If the memory 1001, the processor 1002 and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001 and the processor 1002 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0123] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.

[0124] The processor 1002 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0125] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above method for establishing an eye movement dataset based on multiple traffic operation scenarios.

[0126] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above method for establishing an eye movement dataset based on multiple traffic operation scenarios.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0128] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0129] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0130] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0131] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0132] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0133] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0134] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for establishing an eye movement dataset based on multiple traffic operation scenarios, characterized in that: The following steps are involved: generating eye movement video data of a target driver in the multi-traffic operation scene according to scene video data of the multi-traffic operation scene; Acquire a gaze heat map corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data; Based on the gaze heat map, the semantic segmentation information and three-dimensional reconstruction information of the gaze heat map are obtained by a pre-constructed large-scale direct simultaneous localization and mapping method LSD-SLAM, so as to construct the eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information.

2. The method according to claim 1, characterized in that The obtaining, based on the eye movement video data, a gaze heat map corresponding to each frame of eye movement data in the eye movement video data comprises: Extracting pupil center information of the target driver in each frame of eye movement data; Acquire the sight line estimation information of the target driver in each frame of eye movement data according to the pupil center information; The gaze heat map is obtained based on the line of sight estimation information.

3. The method according to claim 1, characterized in that The method of obtaining semantic segmentation information and three-dimensional reconstruction information of the gaze heat map based on the gaze heat map by a pre-constructed LSD-SLAM, so as to construct an eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information, includes: Generate two-dimensional semantic segmentation information and two-dimensional semantic labels corresponding to each frame of eye movement data based on the SLAM in the pre-built LSD-SLAM; Dividing the gaze heat map by using the two-dimensional semantic label to obtain a first gaze heat map of the gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map; Based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map, and the second semantic segmentation information, obtaining an optimized first gaze heat map corresponding to the first gaze heat map; Based on the optimized first gaze heat map and the first semantic segmentation information, the three-dimensional reconstruction information of the first gaze heat map is obtained in combination with the recursive Bayes and conditional random field in the pre-constructed LSD-SLAM; The eye movement dataset is generated based on the three-dimensional reconstruction information and the optimized first gaze heat map.

4. The method according to claim 1, characterized in that: The step of generating eye movement video data of a target driver in a multi-traffic operation scene according to scene video data of the multi-traffic operation scene comprises: determining key events in the multiple traffic operation scenarios; Extracting scene key video frame data from the scene video data based on the key event; The eye movement video data is determined according to a scene timestamp corresponding to the scene key video frame data.

5. The method according to claim 4, characterized in that The determining the eye movement video data according to the scene timestamp corresponding to the scene key video frame data includes: Acquire the initial eye movement video data of the target driver in the multi-traffic operation scene according to the scene video data; The eye movement video data is determined based on the initial eye movement video data and the scene timestamp.

6. A device for establishing an eye movement dataset based on multiple traffic operation scenarios, characterized in that: include: A generating module, configured to generate eye movement video data of a target driver in the multi-traffic operation scene according to scene video data of the multi-traffic operation scene; An acquisition module, configured to acquire a gaze heat map corresponding to each frame of eye movement data in the eye movement video data based on the eye movement video data; A construction module is used to obtain semantic segmentation information and three-dimensional reconstruction information of the gaze heat map based on the gaze heat map by a pre-constructed LSD-SLAM, so as to construct an eye movement dataset of the target driver based on the semantic segmentation information and the three-dimensional reconstruction information.

7. The device according to claim 6, characterized in that The acquisition module comprises: A first extraction unit is used to extract pupil center information of the target driver in each frame of eye movement data; An acquisition unit, configured to acquire, according to the pupil center information, line of sight estimation information of the target driver in each frame of eye movement data; The first generating unit is used to obtain the gaze heat map based on the line of sight estimation information.

8. The device according to claim 6, characterized in that The building blocks include: A second generating unit is used to generate two-dimensional semantic segmentation information and two-dimensional semantic labels corresponding to each frame of eye movement data based on the SLAM in the pre-built LSD-SLAM; A segmentation unit, configured to segment the gaze heat map using the two-dimensional semantic label to obtain a first gaze heat map of the gaze heat map, first semantic segmentation information corresponding to the first gaze heat map, a second gaze heat map, and second semantic segmentation information corresponding to the second gaze heat map; A third generating unit, configured to obtain an optimized first gaze heat map corresponding to the first gaze heat map based on the first gaze heat map, the first semantic segmentation information, the second gaze heat map, and the second semantic segmentation information; A fourth generating unit, configured to obtain three-dimensional reconstruction information of the first gaze heat map based on the optimized first gaze heat map and the first semantic segmentation information in combination with the recursive Bayes and conditional random field in the pre-constructed LSD-SLAM; A fifth generating unit is used to generate the eye movement data set based on the three-dimensional reconstruction information and the optimized first gaze heat map.

9. The device according to claim 6, characterized in that The generating module comprises: A first determining unit, configured to determine key events in the multiple traffic operation scenarios; A second extraction unit, configured to extract scene key video frame data from the scene video data based on the key event; The second determining unit is used to determine the eye movement video data according to the scene timestamp corresponding to the scene key video frame data.

10. The device according to claim 9, characterized in that The second determining unit includes: an acquisition subunit, configured to acquire initial eye movement video data of the target driver in the multi-traffic operation scene according to the scene video data; A determination subunit is used to determine the eye movement video data based on the initial eye movement video data and the scene timestamp.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for establishing an eye movement data set based on multiple traffic operation scenarios as described in any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for establishing an eye movement data set based on multiple traffic operation scenarios as described in any one of claims 1 to 5.

13. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the method for establishing an eye movement data set based on multiple traffic operation scenarios as described in any one of claims 1 to 5.

Images