A method and system for fusing visual perception information of a person and a vehicle

Abstract

The application relates to a kind of human-vehicle visual perception information fusion method and system, the method comprises: S1, image data acquisition: driver wears eye tracker, with intelligent driving vehicle vehicle-mounted camera synchronous work video, obtain eye tracker video image with fixation point coordinates, and vehicle video image;S2, image data preprocessing: align time frame number, obtain each frame eye tracker image with fixation point coordinates and vehicle image;S3, feature extraction and feature matching: feature point information is extracted from each frame eye tracker image and vehicle image, and multi-scale feature matching algorithm is used to frame by frame matching;S4, image classification fusion: based on the number of matching feature points, driver fixation point is classified and fused in vehicle video image, to obtain the video image containing driver fixation point, to carry out intelligent driving vehicle planning and control.Compared with prior art, the application enhances the perception level of intelligent driving vehicle, improves the safety of automatic driving vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a human-vehicle visual perception information fusion method and system. BACKGROUND

[0002] With the development of autonomous driving technology, the degree of automation in the driving process has been greatly improved. The autonomous driving system can be divided into 6 levels, from level 0 (fully manual driving) to level 5 (fully autonomous driving). However, the current automation level still maintains at the level of partial autonomous driving (L2 or L3). Before achieving full autonomous driving, manual driving and autonomous driving need to work together to ensure driving safety. Therefore, the safety of human-machine co-driving becomes an important challenge for partial autonomous driving.

[0003] The logical framework of the human-machine co-driving system can be divided from bottom to top: the perception layer, the decision layer and the control layer. The perception of the surrounding environment by the intelligent driving vehicle is the premise and basis for decision and control. The current human-machine co-driving solution mainly focuses on the behavior of the driver, such as applying torque on the steering wheel to prevent the driver from turning, giving incentives when the driver is tired, etc.

[0004] However, the existing research rarely involves the research of human-machine co-driving in the field of visual perception. The existing autonomous visual perception algorithm almost predicts targets with the same weight, resulting in differences in detection accuracy of targets of different sizes, and cannot capture key traffic elements in complex traffic scenes. In addition, the current algorithm cannot well simulate the visual features of human driving. Therefore, there is a big obstacle between the autonomous visual perception system and the visual features of the driver. Therefore, a method capable of realizing human-vehicle visual automatic perception information fusion is continuously designed. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a human-vehicle visual perception information fusion method and system, which automatically fuses the fixation point and the vehicle-mounted video image, is efficient, suitable for complex road conditions, has strong scalability, is suitable for different vehicles, and can realize more accurate vehicle perception.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] According to a first aspect of the present application, a human-vehicle visual perception information fusion method is provided, which comprises the following steps:

[0008] Step S1, image data acquisition: the driver wears an eye tracker, and the eye tracker video image with fixation point coordinates and the vehicle-mounted video image are obtained by synchronously working with the vehicle-mounted camera of the intelligent driving vehicle;

[0009] Step S2, image data preprocessing: preprocessing the video image and the fixation point, aligning the time frame number, obtaining the eye tracker image and the vehicle-mounted image of each frame with the fixation point coordinates;

[0010] Step S3, feature extraction and feature matching: extracting feature point information from each frame of eye tracker image and vehicle-mounted image, and performing frame-by-frame matching using a multi-scale feature matching algorithm;

[0011] Step S4, image classification fusion: based on the number of matched feature points, classifying and fusing the driver's fixation point in the vehicle-mounted video image to obtain the vehicle-mounted video image containing the driver's fixation point for planning and control of the intelligent driving vehicle.

[0012] Preferably, the process of obtaining the eye tracker video image with fixation point coordinates in step S1 is specifically:

[0013] A variety of traffic scenarios are set up to construct a fixation point dataset.

[0014] During testing, the intelligent driving car control host clock is used as the reference clock, the eye tracker device timestamp is synchronized, the clock synchronization between multiple sensors is completed, and the eye tracker video image with fixation point coordinates is obtained.

[0015] Preferably, the plurality of traffic scenarios include car-following, lane-changing and overtaking, left / right turning traffic conflicts, conflicts between motor vehicles and pedestrians, and conflicts between motor vehicles and non-motor vehicles.

[0016] Preferably, step S2 is specifically: according to the frame number of the eye tracker and the vehicle-mounted camera, video images are extracted at intervals to complete the time frequency synchronization of multiple sensors, and the fixation point coordinates of the selected frames are calculated using the piecewise linear interpolation method to obtain the eye tracker image and the vehicle-mounted image of each frame with fixation point coordinates.

[0017] Preferably, step S2 further includes filtering the outlying fixation points using Kalman filtering to obtain the eye tracker video image and the vehicle-mounted video image of each frame with fixation point coordinates.

[0018] Preferably, step S3 is specifically:

[0019] Step S31, selecting different scale data to establish a scale space;

[0020] Step S32, establishing the transformation relationship of single peaks in the scale space, and respectively acting on all pixel points of the image under different scale conditions to obtain the transformation values of the pixel points under a specific scale one by one;

[0021] Step S33, analyze the transformation values of all pixel points at a certain scale, if it is an extreme value in the neighborhood, then save the horizontal and vertical coordinates of the pixel point and the current scale in the temporary memory as the preliminary screening information of the image feature point;

[0022] Step S34, repeat step S32 and step S33 in the dimension of sub-pixel, secondary screening of image feature points, and completing the edge position processing;

[0023] Step S35, calculate the main direction of the feature point corresponding to the scale one by one, and store the main direction in the current temporary memory;

[0024] Step S36, calculate the gradient information of each sub-region corresponding to the scale of the feature point one by one, and combine into a descriptor, and store the descriptor in the current temporary memory;

[0025] Step S37, statistics of the feature point horizontal and vertical coordinates, corresponding scale, main direction and descriptor information in the temporary memory, select the nearest descriptor to complete the matching of image feature points.

[0026] Preferably, the step S4 is specifically:

[0027] If the number of matching feature point pairs is less than three, the gaze point cannot be fused, and the gaze point of the frame is determined by linear interpolation of its previous adjacent frame;

[0028] If the number of matching feature point pairs is equal to three, affine transformation is performed to fuse the gaze point.

[0029] If the number of feature points is greater than three, four points are selected for projective transformation, and the gaze point is fused.

[0030] Preferably, if the number of matching feature point pairs is equal to three, affine transformation is performed, and the corresponding transformation expression is:

[0031]

[0032] In the formula, (x, y) is the original pixel point coordinate, (u, v) is the coordinate after affine transformation, (a1, a2) and (b1, b2) are two base vectors in the new coordinate system, and (c1, c2) is the new coordinate origin.

[0033] Preferably, if the number of feature points is greater than three, four points are selected for projective transformation, which is: when selecting feature point pairs, the maximum area is taken as the objective function, and the number of feature point pairs is equal to four as the constraint condition, and the feature point pairs with the best fusion effect are solved.

[0034] The expression of projective transformation is:

[0035] The expression of projective transformation is:

[0036] In the formula, (x, y) are the original pixel coordinates, (x', y') are the coordinates of the point after the projective transformation, w is the new dimension value introduced by homogeneous coordinates into the original pixel coordinates, (x, y, w) represents the homogeneous coordinates of the original pixel; w′ is the new dimension value introduced by homogeneous coordinates into the pixel after the projective transformation, (x′, y′, w′) are the homogeneous coordinates of the point after the projective transformation, and parameter a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 The solution was obtained using the linear least squares method.

[0037] According to a second aspect of the present invention, a human-vehicle visual perception information fusion system is provided, employing any one of the methods described above, the system comprising:

[0038] The human-vehicle perception information acquisition module is used to acquire eye-tracking video images with gaze point coordinates and vehicle-mounted video images;

[0039] The feature extraction and feature matching module is used to perform frame-by-frame matching of feature point information extracted from each frame of eye tracker image and vehicle image using a multi-scale feature matching algorithm;

[0040] The image fusion module is used to classify and fuse the driver's gaze points into the vehicle video image to obtain a vehicle video image containing the driver's gaze points.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1) The method proposed in this invention creatively integrates human attention with vehicle vision, which is highly efficient, applicable to complex road conditions, highly scalable, applicable to different vehicles, and achieves more accurate vehicle perception.

[0043] 2) Multi-scale feature matching is used for feature point matching, and the quantity is classified and fused based on the feature points, which enhances the perception level of intelligent driving vehicles, better supports the subsequent planning and control of autonomous vehicles, and improves the safety of autonomous vehicles.

[0044] 3) The application adopts image classification fusion mode, and the corresponding relationship between the two videos can be accurately solved by finding the feature points corresponding to the vehicle-mounted video and the driver eye tracker visual angle video, so that the fusion has high accuracy, and the vehicle-mounted video and the eye tracker video can both reflect the common and different visual focus after fusion; Specifically, when the number of feature point matching is less than 3, the transformation matrix cannot be solved at this time; At this time, according to the principle of eye fixation smoothness, linear interpolation is selected to ensure the smoothness after fusion; When the number of feature point matching is equal to 3, a unique transformation matrix can be directly solved, and then the image fusion can be carried out; When the number of feature point matching is greater than 3, the transformation matrix can be solved by fitting through the linear least square method, that is, the most possible transformation matrix, and the image can also be accurately fused. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The method flowchart of the application is shown in the figure;

[0046] Figure 2 The method flowchart of the application is shown in the figure;

[0047] Figure 3 The traffic scene schematic diagram in the embodiment is shown in the figure;

[0048] Figure 4 The image shot by the vehicle-mounted camera in the embodiment is shown in the figure;

[0049] Figure 5 The image with fixation points shot by the eye tracker in the embodiment is shown in the figure;

[0050] Figure 6 The image feature point and descriptor matching schematic diagram in the embodiment is shown in the figure;

[0051] Figure 7 The vehicle-mounted image containing the driver fixation points after fusion in the embodiment is shown in the figure. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the application.

[0053] EMBODIMENT

[0054] As shown in the figures, Figure 1 and Figure 2 The embodiment gives a human-vehicle visual perception information fusion method, which comprises the following steps:

[0055] Step S1, image data collection: the driver wears an eye tracker, and the eye tracker works synchronously with the vehicle-mounted camera of the intelligent driving vehicle to record videos, to obtain eye tracker video images with gaze point coordinates and vehicle-mounted video images; wherein the process of obtaining the eye tracker video images with gaze point coordinates is specifically as follows: setting multiple traffic scenes, including following, lane changing, left / right turning traffic conflicts, conflicts between motor vehicles and pedestrians, and conflicts between motor vehicles and non-motor vehicles, to construct a gaze point dataset; during testing, taking the intelligent driving vehicle control host clock as a reference clock, synchronizing the eye tracker device timestamp, completing clock synchronization among multiple sensors, and obtaining the eye tracker video images with gaze point coordinates.

[0056] Step S2, image data preprocessing: preprocessing the video images and gaze points, aligning the time frames, obtaining eye tracker images and vehicle-mounted images with gaze point coordinates for each frame, specifically as follows:

[0057] According to the frame numbers of the eye tracker and the vehicle-mounted camera, the video images are extracted at intervals to complete the time frequency synchronization of multiple sensors, and the segmented linear interpolation method is applied to calculate the gaze point coordinates of the selected frames; the Kalman filter is used to filter the outlying gaze points, to obtain the eye tracker video images and vehicle-mounted video images with gaze point coordinates for each frame.

[0058] Step S3, feature extraction and feature matching: feature point information is extracted from each frame of eye tracker images and vehicle-mounted images, and a multi-scale feature matching algorithm is used for frame-by-frame matching, specifically as follows:

[0059] Step S31, selecting different scale data to establish a scale space;

[0060] Step S32, establishing a transformation relationship of single peaks in the scale space, and respectively acting on all pixel points of the image under different scales to obtain transformation values of the pixel points under a specific scale one by one;

[0061] Step S33, analyzing the transformation values of all pixel points under a specific scale, if it is an extreme value in the neighborhood, then saving the horizontal and vertical coordinate values of the pixel point and the current scale in the temporary memory as the preliminary screening information of the image feature points;

[0062] Step S34, repeating step S32 and step S33 in the dimension of sub-pixels to perform secondary screening of the image feature points, and completing the processing of special positions such as edges;

[0063] Step S35, calculating the main direction of the feature point under the corresponding scale one by one, and storing the main direction in the current temporary memory;

[0064] Step S36, calculating the gradient information of each sub-region corresponding to the feature point under the corresponding scale one by one, and combining them into a descriptor, and storing the descriptor in the current temporary memory;

[0065] Step S37, the characteristic point horizontal and vertical coordinate values in the temporary storage, the corresponding scale, the main direction and the descriptor information are counted, and the nearest descriptor is selected to complete the matching of the image characteristic points.

[0066] Step S4, image classification fusion: based on the number of matched characteristic point pairs, the driver's gaze point is classified and fused in the vehicle-mounted video image to obtain a vehicle-mounted video image containing the driver's gaze point, so as to plan and control the intelligent driving vehicle, specifically:

[0067] 1) if the number of matched characteristic point pairs is less than three, the fusion cannot be performed; the gaze point of the frame is determined by linear interpolation of the previous adjacent frame;

[0068] 2) if the number of matched characteristic point pairs is equal to three, affine transformation is performed, and the corresponding transformation expression is:

[0069]

[0070] In the formula, (x, y) is the original pixel point coordinate, (u, v) is the coordinate after affine transformation, (a1, a2) and (b1, b2) are two base vectors in the new coordinate system, and (c1, c2) is the new coordinate origin.

[0071] 3) if the number of characteristic points is greater than three, four points are selected for projective transformation, specifically: when selecting the characteristic point pairs, the maximum value of the enclosed area is taken as the objective function, and the number of characteristic point pairs is equal to four as the constraint condition, so as to solve the characteristic point pairs with the optimal fusion effect; the expression corresponding to the projective transformation is:

[0072]

[0073] In the formula, (x, y) is the original pixel point coordinate, (x', y') is the coordinate of the point after projective transformation, w is the new dimension value introduced by the original pixel point coordinate into the homogeneous coordinate, (x, y, w) represents the homogeneous coordinate of the original pixel point; w' is the new dimension value introduced by the pixel point after projective transformation into the homogeneous coordinate, (x', y', w') is the homogeneous coordinate of the point after projective transformation, and parameters a 11 , a 12 , a 13 , a 21 , a 22 , a 23 , a 31 , a 32 , a 33 are solved by the linear least square method.

[0074] Next, a system embodiment of the present application is given, a human-vehicle visual perception information fusion system adopts the method, and the system comprises:

[0075] A human-vehicle perception information acquisition module is configured to acquire eye tracker video images with gaze point coordinates and vehicle-mounted video images;

[0076] A feature extraction and feature matching module is configured to perform frame-by-frame matching on feature point information extracted from each frame of eye tracker images and vehicle-mounted images by using a multi-scale feature matching algorithm.

[0077] An image fusion module is configured to fuse the driver's gaze point classification into the vehicle-mounted video images to obtain vehicle-mounted video images containing the driver's gaze point.

[0078] Next, the method and system of the present embodiment will be described in detail.

[0079] (1) According to the computer vision related method, the present embodiment first designs and constructs a real vehicle test, and sets different traffic scenes including following, lane changing, left / right turning traffic conflicts, conflicts between motor vehicles and pedestrians, conflicts between motor vehicles and non-motor vehicles, etc. Experimental data are collected for model training. Next, the image data are processed frame by frame for alignment to prepare for fusion. Then, a multi-scale feature matching algorithm is applied to realize feature point recognition and matching, and the relationship between the eye tracker video images and the vehicle-mounted video images is established. Finally, according to the feature point recognition and matching results, affine transformation or projective transformation is applied to realize image fusion.

[0080] (2) The multi-scale feature matching algorithm used in the feature point recognition and matching of the present invention has the following specific implementation steps: first, different scale data are selected to establish a scale space; second, the transformation relationship of single peaks in the scale space is established, and the transformation values of all pixel points at a specific scale are obtained one by one under the condition of different scales; third, the transformation values of all pixel points at a specific scale are analyzed, if it is an extreme value in the neighborhood, the horizontal and vertical coordinates of the pixel point and the current scale are saved in the temporary memory as the preliminary screening information of the image feature points; fourth, repeat steps two and three in the dimension of sub-pixel, and complete the processing of special positions such as edges; fifth, calculate the main direction of the feature point at the corresponding scale one by one, and store the main direction in the current temporary memory; sixth, calculate the gradient information of each sub-region at the corresponding scale of the feature point one by one, and combine them into a descriptor, and store the descriptor in the current temporary memory; seventh, count the feature point horizontal and vertical coordinates, corresponding scale, main direction and descriptor information in the temporary memory, and select the nearest descriptor to complete the matching of the image feature points.

[0081] (3) The overall algorithm is performed frame by frame. And in the actual application, relevant indicators are set for evaluation, including the reliability of the gaze point, the accuracy of the feature point matching, etc.

[0082] Real vehicle test scene: as shown in Figure 3 The intelligent driving vehicle is set to drive straight, pass through a T-shaped intersection, and enter a vehicle on the left side of the forward direction, and drive away in front of the intelligent driving vehicle, to build a left turn conflict scenario. If the gaze point is on the entering vehicle, it is considered valid.

[0083] Before real vehicle testing, the driver wears an eye tracker on his head, and the gaze focus calibration picture is completed to calibrate the gaze. After starting the recording, the eye data and the front picture will be recorded. The final output data includes timestamp, left and right eye area, height, width, gaze point coordinates, gaze dwell time, saccade dwell time and saccade angle.

[0084] During real vehicle testing, the eye tracker main test machine is connected to the vehicle host through a network line, and the eye tracker device timestamp is synchronized based on the network time protocol, with the intelligent driving car control host clock as the reference clock. The driver's head-mounted eye tracker works simultaneously with the vehicle front camera.

[0085] According to the timestamp alignment video, for 120fps eye tracker video and 30fps vehicle-mounted video images, eye tracker video images are extracted at intervals to make the frame numbers of the two as corresponding as possible. At the same time, the gaze point coordinates corresponding to each eye tracker picture are obtained. If there is no sensor returning the gaze point coordinates at this moment, the coordinates are obtained by linear interpolation. Figure 4 and Figure 5 The vehicle-mounted camera image after the above steps and the eye tracker image with gaze point coordinates.

[0086] A multi-scale feature matching algorithm is used on this group of images to obtain the feature point matching result as shown in Figure 6 .

[0087] According to the number of matched feature point pairs in each group of pictures, the fusion of the gaze point and the vehicle-mounted picture is carried out. If the number of feature point pairs is less than three, the frame gaze point cannot be fused and is determined by linear interpolation of its previous adjacent frame. If the number of feature point pairs is equal to three, affine transformation is performed to fuse the gaze point. If the number of feature point pairs is greater than three, four points are selected for projective transformation. When selecting feature point pairs, the maximum enclosed area is taken as the objective function, and the number of feature point pairs is equal to four as the constraint condition, to solve the feature point pairs with the best fusion effect.

[0088] Figure 4 and Figure 5 Four groups of feature point pairs are obtained after matching, and then the gaze point and the vehicle-mounted picture are fused by projective transformation. The transformation relationship is as follows:

[0089]

[0090] After fusion, the result is as shown in Figure 7The method is used for showing the in-vehicle image containing the driver's gaze point, enhances the intelligent driving vehicle perception level, better supports the subsequent planning and control of the automatic driving vehicle, and improves the safety of the automatic driving vehicle.

[0091] The above merely illustrates the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for fusing human and vehicle visual perception information, characterized in that, The method includes the following steps: Step S1, Image Data Acquisition: The driver wears an eye tracker, which works synchronously with the vehicle's onboard camera to record video, acquiring eye tracker video images with gaze point coordinates and vehicle video images. Step S2, Image Data Preprocessing: Preprocess the video images and fixation points, align the time frame numbers, and obtain eye tracker images and vehicle images with fixation point coordinates for each frame; Step S3, Feature Extraction and Feature Matching: Feature point information is extracted from each frame of the eye tracker image and the vehicle image. A multi-scale feature matching algorithm is used for frame-by-frame matching, specifically: Step S31: Select data at different scales to establish a scale space; Step S32: Establish the transformation relationship of a single peak in the scale space, and apply it to all pixels of the image under different scale conditions to obtain the transformation value of each pixel at a specific scale. Step S33: Analyze the transformation values ​​of all pixels at a specific scale. If it is an extreme value in the neighborhood, save the horizontal and vertical coordinates of the pixel and the current scale in a temporary memory as the initial screening information of image feature points. Step S34: Repeat steps S32 and S33 in the sub-pixel dimension to perform secondary filtering of image feature points and complete edge position processing. Step S35: Calculate the principal direction at the corresponding scale for each feature point and store the principal direction in the current temporary memory; Step S36: Calculate the gradient information of each sub-region at the scale corresponding to each feature point, combine them into a descriptor, and store the descriptor in the current temporary memory; Step S37: Calculate the horizontal and vertical coordinates, corresponding scales, main directions, and descriptor information of the feature points in the temporary storage, and select the nearest descriptor to complete the matching of image feature points; Step S4, Image Classification and Fusion: Based on the number of matching feature point pairs, the driver's gaze points are classified and fused into the in-vehicle video image to obtain an in-vehicle video image containing the driver's gaze points, for planning and control of intelligent driving vehicles. Specifically: If the number of matching feature point pairs is less than three, fusion is not possible, and the gaze point of that frame is determined by linear interpolation of its preceding neighboring frame. If the number of matching feature point pairs is equal to three, then an affine transformation is performed to fuse the gaze points; If the number of feature points is greater than three, then four points are selected for projective transformation to fuse the gaze points.

2. The method for fusing human and vehicle visual perception information according to claim 1, characterized in that, The process of acquiring the eye-tracking video image with fixation point coordinates in step S1 is as follows: Set up various traffic scenarios and construct a gaze point dataset; During testing, the clock of the intelligent driving vehicle's control host was used as a reference clock to synchronize the timestamps of the eye tracker devices, thereby achieving clock synchronization among multiple sensors and acquiring eye tracker video images with gaze coordinates.

3. The method for fusing human and vehicle visual perception information according to claim 2, characterized in that, The various traffic scenarios include following other vehicles, lane changing and overtaking, left / right turn traffic conflicts, conflicts between motor vehicles and pedestrians, and conflicts between motor vehicles and non-motor vehicles.

4. The method for fusing human-vehicle visual perception information according to claim 1, characterized in that, Step S2 specifically involves: extracting video images at intervals based on the frame counts of the eye tracker and the vehicle camera to achieve multi-sensor time-frequency synchronization, and calculating the gaze coordinates of the selected frames using a piecewise linear interpolation method to obtain an eye tracker image and a vehicle image with gaze coordinates for each frame.

5. The method for fusing human-vehicle visual perception information according to claim 4, characterized in that, Step S2 further includes using Kalman filtering to filter outlier gaze points, resulting in eye-tracking video images and vehicle-mounted video images for each frame containing gaze point coordinates.

6. The method for fusing human and vehicle visual perception information according to claim 1, characterized in that, If the number of matching feature point pairs is equal to three, then an affine transformation is performed, and the corresponding transformation expression is: In the formula, These are the original pixel coordinates. The coordinates are after affine transformation. , These are two basis vectors in the new coordinate system. Let this be the new origin of the coordinate system.

7. The method for fusing human and vehicle visual perception information according to claim 1, characterized in that, If the number of feature points is greater than three, then four points are selected for projective transformation, specifically as follows: When selecting feature point pairs, the objective function is the maximum value of the enclosing area, and the constraint is that the number of feature point pairs is equal to four. The optimal feature point pair for fusion effect is then determined. The expression corresponding to the projective transformation is: In the formula, These are the original pixel coordinates. Let these be the coordinates of the point after the projective transformation. Introduce a new dimension value, homogeneous coordinates, to the original pixel coordinates. Represents the homogeneous coordinates of the original pixel; This introduces a new dimension value, homogeneous coordinates, to the pixels after projective transformation. Let be the homogeneous coordinates of the point after the projective transformation, and be the parameters. , , , , , , , , The solution was obtained using the linear least squares method.

8. A human-vehicle visual perception information fusion system, characterized in that, The system, employing the method according to any one of claims 1 to 7, comprises: The human-vehicle perception information acquisition module is used to acquire eye-tracking video images with gaze point coordinates and vehicle-mounted video images; The feature extraction and feature matching module is used to perform frame-by-frame matching of feature point information extracted from each frame of eye tracker image and vehicle image using a multi-scale feature matching algorithm; The image fusion module is used to classify and fuse the driver's gaze points into the vehicle video image to obtain a vehicle video image containing the driver's gaze points.

Images