A sight calibration method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a line of sight calibration method, apparatus, electronic device, and storage medium. The electronic device can obtain target line of sight data captured by a cockpit camera; based on the coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration marker in the projection area, determine the coordinates of the calibration marker in the cockpit camera coordinate system as the predicted line of sight landing point coordinates; based on the target line of sight data, determine the target line of sight landing point coordinates in the cockpit camera coordinate system; and determine line of sight calibration parameters based on the target line of sight landing point coordinates and the predicted line of sight landing point coordinates. Since the coordinates of the calibration marker in the projection area are known, the electronic device can accurately determine the coordinates of the calibration marker in the cockpit camera coordinate system, i.e., the predicted line of sight landing point coordinates. Furthermore, the calibration parameters can be accurately determined based on the predicted line of sight landing point coordinates and the target line of sight landing point coordinates, thereby achieving line of sight calibration with high accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of line of sight calibration, and in particular to a line of sight calibration method, device, electronic device and storage medium. Background Art

[0002] Most traffic accidents today are related to driver fatigue or distraction, leading to the widespread use of driver monitoring systems (DMS). These systems track the driver's gaze in real time to determine if they are experiencing driver fatigue or distraction. If these conditions occur, they can alert the driver through voice notifications, thereby reducing the risk of accidents.

[0003] The accuracy of gaze tracking depends on the accuracy of gaze calibration. Current gaze calibration methods typically involve directing the driver's gaze at certain fixed points, capturing gaze data from those fixed points, and then determining calibration parameters based on the gaze data and the predicted gaze data corresponding to those fixed points. However, in the actual cockpit environment, the coordinates of fixed points are difficult to accurately determine, making gaze calibration difficult. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a line of sight calibration method, device, electronic device, and storage medium to achieve line of sight calibration. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a line of sight calibration method, the method comprising:

[0006] Acquiring target sight data of a user toward a calibration marker captured by a cockpit camera, wherein the calibration marker is displayed in a projection area of ​​the cockpit, the calibration marker being used to indicate a location at which the user is looking at the calibration marker, and the projection area being an area in front of the cockpit for displaying projection information;

[0007] Based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration mark in the projection area, determining the coordinates of the calibration mark in the cockpit camera coordinate system as the predicted sight point coordinates;

[0008] Determining the target sight point coordinates of the user with respect to the calibration mark in the cockpit camera coordinate system based on the target sight line data;

[0009] Determine the sight line calibration parameters according to the target sight line landing point coordinates and the predicted sight line landing point coordinates.

[0010] Optionally, before the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method further includes:

[0011] Acquire a first target image of the projection area captured by the cockpit camera and reflected by a plane mirror;

[0012] Determining coordinates of the reflected projection area in a world coordinate system based on pre-calibrated parameters of the cockpit camera and pixel coordinates of the projection area in the first target image;

[0013] Determine the coordinates of the projection area in the world coordinate system according to the distance between the projection area and the plane mirror, the coordinates of the reflected projection area in the world coordinate system, and the plane reflection imaging principle;

[0014] According to a pre-calibrated conversion relationship between the world coordinate system and the cockpit camera coordinate system, the coordinates of the projection area in the world coordinate system are converted into the coordinates of the projection area in the cockpit camera coordinate system.

[0015] Optionally, before the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method further includes:

[0016] Acquire a second target image of the projection area showing the first target captured by the cockpit camera and reflected by the plane mirror;

[0017] determining the coordinates of the reflected projection area in the cockpit camera coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the first target in the second target image;

[0018] Acquiring a third target image of the plane mirror with a second target attached to the surface thereof, captured by the cockpit camera;

[0019] Determining the coordinates of the plane mirror in the cockpit camera coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the second target in the third target image;

[0020] The coordinates of the projection area in the cockpit camera coordinate system are determined according to the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the plane reflection imaging principle.

[0021] Optionally, the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera includes:

[0022] Control the projection area of ​​the cockpit to display calibration marks and record the number of calibration marks displayed;

[0023] Acquiring user's sight line data with respect to the displayed calibration mark captured by the cockpit camera;

[0024] determining whether the number of recorded displayed calibration marks reaches a preset number;

[0025] If the recorded number of displayed calibration marks does not reach the preset number, returning to the projection area of ​​the control cabin to display the calibration marks and recording the number of displayed calibration marks;

[0026] If the number of the recorded displayed calibration marks reaches the preset number, the acquired sight line data is determined as the user's target sight line data for the calibration marks.

[0027] Optionally, before the step of determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight data, the method further includes:

[0028] performing data cleaning on the target sight line data to obtain cleaned target sight line data;

[0029] The step of determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight data includes:

[0030] Based on the cleaned target sight line data, the coordinates of the user's target sight line landing point for the calibration mark are determined in the cockpit camera coordinate system.

[0031] Optionally, the sight line data includes a plurality of sight line position coordinates of the user captured by a cockpit camera within a preset time period after the calibration mark is displayed;

[0032] The step of performing data cleaning on the target sight line data to obtain cleaned target sight line data includes performing data cleaning in one of the following ways to obtain cleaned target sight line data:

[0033] Clustering the plurality of sight line position coordinates to determine a cluster center; and determining the cluster center as the cleaned target sight line data;

[0034] Clustering the plurality of sight line position coordinates to determine a category including the largest number of sight line position coordinates; and determining the sight line position coordinates included in the determined category as cleaned target sight line data;

[0035] Determining a user's sight focus set based on a capture time sequence corresponding to the plurality of sight position coordinates; determining an end point position coordinate of the sight focus set as cleaned target sight data;

[0036] Based on the capture timing corresponding to the multiple line of sight position coordinates, the user's line of sight focus set is determined; the line of sight position coordinates whose distance from the end position coordinates of the line of sight focus set is less than a preset threshold are determined as the cleaned target line of sight data.

[0037] Optionally, the step of determining the sight line calibration parameters according to the difference between the target sight line landing point coordinates and the predicted sight line landing point coordinates includes:

[0038] Based on the target sight point coordinates and the predicted sight point coordinates, the sight calibration matrix R is calculated according to the following formula:

[0039]

[0040] Where n is the number of target sight point coordinates, V3 ′ ×n V is a matrix whose column vectors are composed of the components of the predicted sight point coordinates. n ′ ×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors consist of the components of the coordinates of the target sight point.

[0041] Optionally, after the step of determining the sight line calibration parameters based on the difference between the target sight line landing point coordinates and the predicted sight line landing point coordinates, the method further includes:

[0042] Obtaining the user's current sight line data;

[0043] The product of the sight line calibration matrix and the current sight line data is determined as the calibrated sight line data.

[0044] Optionally, before the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method further includes:

[0045] Controlling the cockpit's projection area to display sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area, and / or,

[0046] The cockpit audio output device is controlled to play a sight line calibration prompt voice, wherein the sight line calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area.

[0047] Optionally, the calibration mark is displayed in a projection area of ​​a head-up display (HUD) in the cockpit.

[0048] In a second aspect, an embodiment of the present application provides a sight line calibration device, the device comprising:

[0049] a sight line data acquisition module, configured to acquire target sight line data of a user directed at a calibration marker, as captured by a cockpit camera; wherein the calibration marker is displayed in a projection area of ​​the cockpit and indicates the location of the calibration marker to be gazed at by the user; and the projection area is an area in front of the cockpit for displaying projection information;

[0050] a predicted sight line coordinate determination module, configured to determine the coordinates of the calibration marker in the cockpit camera coordinate system as the predicted sight line landing point coordinates based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration marker in the projection area;

[0051] a target sight line coordinate determination module, configured to determine the coordinates of the user's target sight line landing point for the calibration mark in the cockpit camera coordinate system based on the target sight line data;

[0052] The sight line calibration parameter determination module is used to determine the sight line calibration parameters according to the target sight line landing point coordinates and the predicted sight line landing point coordinates.

[0053] Optionally, the device further includes:

[0054] A first target image acquisition module is used to acquire a first target image of the projection area captured by the cockpit camera after being reflected by the plane mirror;

[0055] a first projection area coordinate determination module, configured to determine the coordinates of the projection area after reflection in a world coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the projection area in the first target image;

[0056] a second projection area coordinate determination module, configured to determine the coordinates of the projection area in the world coordinate system according to the distance between the projection area and the plane mirror, the coordinates of the reflected projection area in the world coordinate system, and the plane reflection imaging principle;

[0057] a third projection area coordinate determination module, configured to convert the coordinates of the projection area in the world coordinate system into the coordinates of the projection area in the cockpit camera coordinate system according to a pre-calibrated conversion relationship between the world coordinate system and the cockpit camera coordinate system;

[0058] The device further comprises:

[0059] A second target image acquisition module is used to acquire a second target image of the projection area showing the first target captured by the cockpit camera after being reflected by the plane mirror;

[0060] a fourth projection area coordinate determination module, configured to determine the coordinates of the reflected projection area in the cockpit camera coordinate system based on pre-calibrated cockpit camera parameters and the pixel coordinates of the first target in the second target image;

[0061] A third target image acquisition module is used to acquire a third target image of the plane mirror with the second target attached to the surface captured by the cockpit camera;

[0062] a plane mirror coordinate determination module, configured to determine the coordinates of the plane mirror in the cockpit camera coordinate system based on pre-calibrated parameters of the cockpit camera and the pixel coordinates of the second target in the third target image;

[0063] a fifth projection area coordinate determination module, configured to determine the coordinates of the projection area in the cockpit camera coordinate system based on the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the plane reflection imaging principle;

[0064] The target sight line coordinate determination module includes:

[0065] A first calibration mark display submodule, configured to control the projection area of ​​the cockpit to display the calibration mark and record the number of displayed calibration marks;

[0066] a sight line data acquisition submodule, configured to acquire sight line data of the user with respect to the displayed calibration mark captured by the cockpit camera;

[0067] The quantity determination submodule is used to determine whether the number of recorded and displayed calibration marks reaches a preset number;

[0068] a second calibration mark display submodule, configured to return to the projection area of ​​the control cabin to display the calibration mark and record the number of displayed calibration marks if the number of displayed calibration marks does not reach the preset number;

[0069] a target sight line data determination submodule, configured to determine the acquired sight line data as the user's target sight line data for the calibration mark if the number of recorded displayed calibration marks reaches the preset number;

[0070] The device further comprises:

[0071] A data cleaning module, configured to clean the target sight line data to obtain cleaned target sight line data;

[0072] The target sight line coordinate determination module includes:

[0073] A target sight point coordinate submodule is configured to determine the target sight point coordinates of the user for the calibration mark in the cockpit camera coordinate system based on the cleaned target sight data;

[0074] The sight line data includes a plurality of sight line position coordinates of the user captured by the cockpit camera within a preset time period after the calibration mark is displayed;

[0075] The data cleaning module includes:

[0076] A first cleaning submodule is configured to cluster the plurality of sight line position coordinates to determine a cluster center; and determine the cluster center as the cleaned target sight line data;

[0077] A second cleaning submodule is configured to cluster the plurality of sight line position coordinates to determine a category including the largest number of sight line position coordinates; and determine the sight line position coordinates included in the determined category as cleaned target sight line data;

[0078] A third cleaning submodule is configured to determine a sight focus set of the user based on a capture time sequence corresponding to the plurality of sight position coordinates; and determine the end point position coordinates of the sight focus set as the cleaned target sight data;

[0079] a fourth cleaning submodule, configured to determine a sight focus set of the user based on a capture time sequence corresponding to the plurality of sight position coordinates; and determine a sight position coordinate whose distance from an end position coordinate of the sight focus set is less than a preset threshold as the cleaned target sight data;

[0080] The sight line calibration parameter determination module includes:

[0081] The calibration matrix calculation submodule is used to calculate the sight calibration matrix R based on the target sight point coordinates and the predicted sight point coordinates according to the following formula:

[0082]

[0083] Where n is the number of target sight point coordinates, V3 ′ ×n V is a matrix whose column vectors are composed of the components of the predicted sight point coordinates. n ′ ×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors are composed of the components of the coordinates of the target sight point;

[0084] The device further comprises:

[0085] A current sight line data acquisition module, configured to acquire the current sight line data of the user;

[0086] a sight line data calibration module, configured to determine the product of the sight line calibration matrix and the current sight line data as calibrated sight line data;

[0087] The device further comprises:

[0088] A sight calibration prompt information display module, used to control the projection area of ​​the cockpit to display a sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area;

[0089] The sight calibration prompt voice playing module is used to control the cockpit audio output device to play the sight calibration prompt voice, wherein the sight calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area.

[0090] The calibration mark is displayed in a projection area of ​​a head-up display (HUD) in the cockpit.

[0091] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0092] Memory for storing computer programs;

[0093] The processor is configured to implement any of the methods described in the first aspect above when executing a program stored in the memory.

[0094] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the methods described in the first aspect above.

[0095] Beneficial effects of the embodiments of the present application:

[0096] In the solution provided by an embodiment of the present application, an electronic device can obtain target gaze data of a user toward a calibration marker captured by a cabin camera. The calibration marker is displayed in a projection area of ​​the cabin, indicating the location of the calibration marker to which the user is looking. The projection area is the area located in front of the cabin and displays projected information. The electronic device determines the coordinates of the calibration marker in the cabin camera coordinate system based on pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration marker in the projection area, as predicted gaze landing point coordinates. The electronic device determines the coordinates of the user's target gaze landing point toward the calibration marker in the cabin camera coordinate system based on the target gaze data. The electronic device determines gaze calibration parameters based on the target gaze landing point coordinates and the predicted gaze landing point coordinates. Because the electronic device can control the projection area to display the calibration marker, and the coordinates of the calibration marker in the projection area are known, the electronic device can accurately determine the coordinates of the calibration marker in the cabin camera coordinate system, i.e., the predicted gaze landing point coordinates, based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration marker in the projection area. In this way, the electronic device can accurately determine the calibration parameters based on the difference between the predicted gaze landing point coordinates and the target gaze landing point coordinates, thereby achieving gaze calibration with high accuracy.

[0097] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0099] Figure 1 A flow chart of a sight line calibration method provided in an embodiment of the present application;

[0100] Figure 2 Based on Figure 1 A schematic diagram of a calibration mark displayed in the projection area of ​​the illustrated embodiment;

[0101] Figure 3 Based on Figure 1 A flow chart of a method for determining the coordinates of the projection area in the cockpit camera coordinate system of the illustrated embodiment;

[0102] Figure 4 Based on Figure 1 A schematic diagram of the projection area of ​​the embodiment shown being imaged by reflection from a plane mirror;

[0103] Figure 5 Based on Figure 1 Another flow chart of a method for determining the coordinates of the projection area in the cockpit camera coordinate system of the illustrated embodiment;

[0104] Figure 6 for Figure 1 A specific flow chart of step S101 in the embodiment shown;

[0105] Figure 7 Based on Figure 1 A flow chart of the sight line calibration method of the illustrated embodiment;

[0106] Figure 8 Based on Figure 1 A flow chart of a method for determining sight line calibration parameters according to the illustrated embodiment;

[0107] Figure 9 A schematic structural diagram of a sight line calibration device provided in an embodiment of the present application;

[0108] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0109] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0110] To achieve line of sight calibration, embodiments of the present application provide a line of sight calibration method, apparatus, electronic device, computer-readable storage medium, and computer program product. A line of sight calibration method provided by embodiments of the present application is first introduced below.

[0111] The sight line calibration method provided in the embodiments of the present application can be applied to any electronic device that requires sight line calibration, for example, a cockpit processor, a sight line calibration device, etc., without specific limitation herein. For clarity of description, it will be referred to as an electronic device in the following text.

[0112] like Figure 1 As shown, a line of sight calibration method comprises:

[0113] S101, obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera;

[0114] The calibration mark is displayed in the projection area of ​​the cabin, and the calibration mark is used to instruct the user to look at the location of the calibration mark. The projection area is an area located in front of the cabin for displaying projection information.

[0115] S102, based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration mark in the projection area, determining the coordinates of the calibration mark in the cockpit camera coordinate system as the predicted sight point coordinates;

[0116] S103, determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight line data;

[0117] S104: Determine sight line calibration parameters according to the target sight line landing point coordinates and the predicted sight line landing point coordinates.

[0118] As can be seen, in the solution provided by the embodiment of the present application, the electronic device can obtain the target sight line data of the user toward the calibration mark captured by the cabin camera, wherein the calibration mark is displayed in the projection area of ​​the cabin and is used to indicate the location of the calibration mark at which the user is looking. The projection area is the area located in front of the cabin for displaying projected information. Based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration mark in the projection area, the coordinates of the calibration mark in the cabin camera coordinate system are determined as the predicted sight line landing point coordinates. Based on the target sight line data, the coordinates of the user's target sight line landing point toward the calibration mark in the cabin camera coordinate system are determined. The sight line calibration parameters are determined based on the target sight line landing point coordinates and the predicted sight line landing point coordinates. Since the electronic device can control the projection area to display the calibration mark, and the coordinates of the calibration mark in the projection area are known, the electronic device can accurately determine the coordinates of the calibration mark in the cabin camera coordinate system, i.e., the predicted sight line landing point coordinates, based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration mark in the projection area. In this way, the electronic device can accurately determine the calibration parameters according to the difference between the predicted sight point coordinates and the target sight point coordinates, thereby realizing sight calibration with high accuracy.

[0119] The projection area in the embodiments of the present application may be a projection area corresponding to a head-up display (HUD). The HUD can project information that the user would otherwise need to look down at onto the projection area in front of the cabin. The projection area can be used to display projection information, which may include navigation information, incoming call information, and social media app information, without specific limitation.

[0120] In one embodiment, the head-up display can project information onto a physical carrier in front of the cabin, such as a translucent resin plate. In this case, the projection area is the physical area on the physical carrier where the projection information is displayed. In another embodiment, the head-up display can project information onto a virtual plane in front of the cabin. In this case, the projection area is the virtual area in front of the cabin where the projection information is displayed.

[0121] The electronic device can control the projection area to display a calibration mark. When the user looks at the calibration mark, the cockpit camera can capture the user's target line of sight, thereby executing step S101. The calibration mark can be any pattern that indicates the user's gaze position and can be configured based on line of sight calibration requirements. Parameters such as the shape, display position, color, and size of the calibration mark are not specifically limited.

[0122] The schematic diagram of the calibration mark displayed in the projection area of ​​the cockpit can be as follows: Figure 2 As shown, the calibration mark 201 is a black dot, and the calibration mark 201 is displayed in the projection area 202. The projection area 202 is located in front of the cockpit. The projection area 202 and the driver's eyeball 203 are respectively located on both sides of the front windshield 204. The cockpit camera 205 can capture the target line of sight data of the driver's eyeball 203 for the calibration mark 201.

[0123] In the embodiment of the present application, the format of the target sight line data is not specifically limited. For example, the target sight line data can be an image captured by a cabin camera when the user is looking at the calibration mark, the coordinates of the sight line direction when the user is looking at the calibration mark, etc.

[0124] Since the cockpit camera is facing the user and the projection area is located in front of the cockpit, the projection area cannot be captured by the cockpit camera. Therefore, the calibration mark displayed in the projection area cannot be captured by the cockpit camera. Therefore, in order to determine the coordinates of the calibration mark displayed in the projection area in the cockpit camera coordinate system, the coordinates of the projection area in the cockpit camera coordinate system can be calibrated in advance.

[0125] In this way, since the coordinates of the calibration mark in the projection area are known, the electronic device can determine the coordinates of the calibration mark in the cockpit camera coordinate system based on the coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration mark in the projection area as the predicted line of sight landing point coordinates, that is, execute step S102.

[0126] If the user's line of sight does not deviate, then when the user looks at a calibration mark, the position the user should look at is the above-mentioned predicted line of sight landing point coordinates. Therefore, the predicted line of sight landing point coordinates can be used to represent the line of sight direction of the user when the user looks at the calibration mark when the user's line of sight does not deviate.

[0127] In order to determine the deviation between the coordinates of the user's actual line of sight direction when looking at the calibration mark and the predicted line of sight landing point coordinates, the electronic device can determine the coordinates of the user's target line of sight landing point for the calibration mark in the cockpit camera coordinate system based on the acquired target line of sight data, that is, execute step S103.

[0128] In one embodiment, the target gaze data is an image captured by a cabin camera of the user gazing at the calibration mark. The electronic device can identify the user's eye position and eye direction in the image and, based on the cabin camera's intrinsic parameters, determine the transformation relationship between the cabin camera coordinate system and the image coordinate system. Furthermore, based on this transformation relationship and the coordinates corresponding to the user's eye position and eye direction in the image, the electronic device can determine the eye position and eye direction in the cabin camera coordinate system. The eye direction is a ray in the cabin camera coordinate system. The coordinates of the eye position in the cabin camera coordinate system can also be obtained using a depth camera. Next, the electronic device can determine the intersection of the eye direction in the cabin camera coordinate system and the projection plane, and determine the coordinates of this intersection as the coordinates of the user's target gaze point with respect to the calibration mark.

[0129] In another embodiment, the target gaze data is the eye position and eye direction of the user when looking at the calibration mark, as captured by a head-mounted gaze capture device. For example, the head-mounted gaze capture device may be an eye-tracking device. The eye tracker captures the eye position and eye direction in the eye tracker coordinate system. The electronic device can convert the eye position and eye direction from the eye tracker coordinate system to the cockpit camera coordinate system based on a pre-calibrated conversion relationship between the eye tracker coordinate system and the cockpit camera coordinate system. In this way, the electronic device can determine the intersection of the eye direction in the cockpit camera coordinate system and the projection plane, and determine the coordinates of the intersection as the coordinates of the user's target gaze landing point on the calibration mark.

[0130] For the same calibration mark, the target sight point coordinates represent the actual sight direction of the user when gazing at the calibration mark, and the predicted sight point coordinates represent the predicted sight direction of the user when gazing at the calibration mark, assuming no deviation in the user's sight. Therefore, the electronic device can determine the sight calibration parameters based on the target sight point coordinates and the predicted sight point coordinates, thus executing step S104.

[0131] The sight line calibration parameters are used to calibrate the user's actual sight line data to correct any deviations in the user's sight line. In a first embodiment, the electronic device may calculate a mapping relationship between the target sight line landing point coordinates and the predicted sight line landing point coordinates, and then use the mapping relationship as the sight line calibration parameters.

[0132] In a second embodiment, the electronic device can determine the target gaze direction vector of the user gazing at the calibration marker based on the target gaze point coordinates and the user's eye position. Furthermore, based on the predicted gaze point coordinates and the user's eye position, the electronic device can determine the predicted gaze direction vector of the user gazing at the calibration marker, assuming no deviation in the user's gaze. In this way, the electronic device can determine the gaze calibration parameters based on the difference between the target gaze direction vector and the predicted gaze direction vector.

[0133] In a third embodiment, the electronic device can determine the target gaze direction vector of the user based on the target gaze point coordinates and the user's eye position; and based on the predicted gaze point coordinates and the user's eye position, determine the predicted gaze direction vector of the user gazing at the calibration mark, assuming no deviation in the user's gaze. In this way, the electronic device can determine gaze calibration parameters based on the user's eye position and the target gaze direction vector, the user's eye position, and the predicted gaze direction vector.

[0134] As can be seen, in this embodiment of the present application, since the electronic device can control the projection area to display the calibration mark, and the coordinates of the calibration mark in the projection area are known, the electronic device can accurately determine the coordinates of the calibration mark in the cockpit camera coordinate system based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration mark in the projection area, i.e., the predicted line of sight landing point coordinates. In this way, the electronic device can accurately determine the calibration parameters based on the difference between the predicted line of sight landing point coordinates and the target line of sight landing point coordinates, thereby achieving line of sight calibration with high accuracy.

[0135] As an implementation method of the present application, Figure 3 As shown, before the step of obtaining the target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method may further include:

[0136] S301, obtaining a first target image of the projection area captured by the cockpit camera and reflected by a plane mirror;

[0137] Since the projection area is in front of the cockpit and the cockpit camera is facing the direction of the user's driving seat, in order to enable the cockpit camera to indirectly capture the projection area, a plane mirror can be installed in the cockpit so that the image formed by the projection area after reflection by the plane mirror is located in the direction of the user's driving seat. In this way, the cockpit camera can capture the first target image of the image, that is, the first target image of the projection area after reflection by the plane mirror, and then the electronic device can obtain the first target image captured by the cockpit camera.

[0138] The schematic diagram of the projection area reflected by the plane mirror can be shown as follows: Figure 4 As shown, projection area 401 is located in front of the cockpit, and cockpit camera 402 is oriented toward the driver's seat in the cockpit. An image 404 formed by reflection of projection area 401 by plane mirror 403 is located in the direction of the driver's seat in the cockpit, and cockpit camera 402 can capture a first target image of image 404.

[0139] S302, determining the coordinates of the reflected projection area in a world coordinate system based on pre-calibrated parameters of the cockpit camera and the pixel coordinates of the projection area in the first target image;

[0140] To determine the transformation relationship between the pixel coordinate system and the world coordinate system, the cockpit camera's intrinsic parameters can be pre-calibrated. These parameters may include the camera's scale factor in the two coordinate axes of the image coordinate system, the principal point coordinates relative to the image coordinate system, the focal length, the optical center position, the tilt parameters of the image coordinate system's coordinate axes, and distortion parameters. In one embodiment, the cockpit camera's intrinsic parameters can be determined using the Zhang Zhengyou calibration method.

[0141] In this way, the electronic device can determine the coordinates of the reflected projection area in the world coordinate system according to the parameters of the cockpit camera and the pixel coordinates of the projection area in the first target image.

[0142] S303, determining the coordinates of the projection area in the world coordinate system according to the distance between the projection area and the plane mirror, the coordinates of the reflected projection area in the world coordinate system, and the plane reflection imaging principle;

[0143] Because the projection area is reflected by the plane mirror to form a post-reflection projection area, the post-reflection projection area is the same distance from the projection area to the plane mirror. Once the coordinates of the post-reflection projection area in the world coordinate system have been determined, the electronic device can determine the coordinates of the projection area in the world coordinate system based on the coordinates of the projection area in the world coordinate system, the distance between the projection area and the plane mirror, and the principle of plane reflection imaging.

[0144] S304 , converting the coordinates of the projection area in the world coordinate system into the coordinates of the projection area in the cockpit camera coordinate system according to a pre-calibrated conversion relationship between the world coordinate system and the cockpit camera coordinate system.

[0145] To determine the coordinates of the projected area in the cockpit camera coordinate system, the transformation relationship between the cockpit camera coordinate system and the world coordinate system can be pre-calibrated. Specifically, the cockpit camera's extrinsic parameters can be pre-calibrated. These extrinsic parameters can include translation vectors and rotation matrices. As an implementation, the cockpit camera's extrinsic parameters can be determined using the Zhang Zhengyou calibration method.

[0146] In this way, the electronic device can convert the coordinates of the projection area in the world coordinate system into the coordinates of the projection area in the cockpit camera coordinate system according to the conversion relationship between the world coordinate system and the cockpit camera coordinate system.

[0147] As can be seen, in this embodiment of the present application, since the projection area is located in front of the cabin and the cabin camera faces the direction of the user's driving position, a plane mirror can be installed in the cabin. The cabin camera can capture the first target image of the projection area after it is reflected by the plane mirror. To determine the transformation relationship between the pixel coordinate system and the world coordinate system, and between the world coordinate system and the cabin camera coordinate system, the cabin camera's intrinsic and extrinsic parameters can be pre-calibrated. In this way, the electronic device can determine the coordinates of the projection area in the world coordinate system after reflection based on the pixel coordinates of the projection area in the first target image and the cabin camera's intrinsic parameters. The electronic device can determine the coordinates of the projection area in the world coordinate system based on the distance between the projection area and the plane mirror, the coordinates of the projection area in the world coordinate system after reflection, and the principle of planar reflection imaging. Furthermore, based on the transformation relationship between the world coordinate system and the cabin camera coordinate system, the coordinates of the projection area in the world coordinate system are converted to the coordinates of the projection area in the cabin camera coordinate system. This allows the accurate determination of the coordinates of the projection area in the cabin camera coordinate system, thereby achieving line of sight calibration.

[0148] As an implementation method of the present application, Figure 5 As shown, before the step of obtaining the target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method may further include:

[0149] S501, obtaining a second target image of the projection area showing the first target captured by the cockpit camera and reflected by a plane mirror;

[0150] Since the projection area is in front of the cockpit and the cockpit camera is facing the direction of the driving seat where the user is located, in order to enable the cockpit camera to indirectly capture the projection area, a plane mirror can be set up in the cockpit and the position of the plane mirror can be fixed so that the image formed by the projection area after being reflected by the plane mirror is located in the direction of the driving seat where the user is located. In this way, the cockpit camera can capture a second target image of the image, that is, the second target image of the projection area after being reflected by the plane mirror, and then the electronic device can obtain the second target image captured by the cockpit camera.

[0151] The electronic device can control the projection area to display a first target. The first target can be one of a checkerboard pattern, a circular pattern, and a random pattern, without specific limitation herein. In this way, the electronic device can determine the coordinates of the first target in the cockpit camera coordinate system. Since the first target is displayed in the projection area, the coordinates of the first target in the cockpit camera coordinate system can be regarded as the coordinates of the projection area in the cockpit camera coordinate system.

[0152] S502, determining the coordinates of the reflected projection area in the cockpit camera coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the first target in the second target image;

[0153] To determine the transformation relationship between the image coordinate system and the cockpit camera coordinate system, the cockpit camera's intrinsic parameters can be pre-calibrated. These parameters may include the cockpit camera's scale factor along the two coordinate axes of the image coordinate system, the coordinates of the principal point relative to the image coordinate system, the focal length, the position of the optical center, the tilt parameters of the image coordinate system's coordinate axes, and distortion parameters. As an implementation, the cockpit camera's intrinsic parameters can be determined using the Zhang Zhengyou calibration method.

[0154] In this way, the electronic device can determine the coordinates of the reflected projection area in the cockpit camera coordinate system based on the cockpit camera parameters and the pixel coordinates of the first target in the second target image.

[0155] S503, acquiring a third target image of the plane mirror with a second target attached to its surface, captured by the cockpit camera;

[0156] After obtaining the coordinates of the reflected projection area in the cockpit camera coordinate system, since the reflected projection area is formed by the projection area being reflected by the plane mirror, in order to determine the coordinates of the projection area in the cockpit camera coordinate system, the electronic device can first determine the coordinates of the plane mirror in the cockpit camera coordinate system.

[0157] Specifically, the electronic device can acquire a third target image captured by the cockpit camera of the plane mirror having a second target attached to its surface. The second target can be a checkerboard pattern, a circular pattern, or a random pattern. This allows the electronic device to determine the coordinates of the second target in the cockpit camera coordinate system. Since the second target is attached to the surface of the plane mirror, the coordinates of the second target in the cockpit camera coordinate system can be considered the coordinates of the plane mirror in the cockpit camera coordinate system.

[0158] S504, determining the coordinates of the plane mirror in the cockpit camera coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the second target in the third target image;

[0159] In order to determine the coordinates of the plane mirror in the cockpit camera coordinate system, and then determine the coordinates of the projection area in the cockpit camera coordinate system, the electronic device can determine the coordinates of the plane mirror in the cockpit camera coordinate system based on the pre-calibrated internal parameters of the cockpit camera and the pixel coordinates of the second target in the third target image.

[0160] As an embodiment, if the cockpit camera simultaneously captures the second and third target images, the position of the plane mirror does not need to be fixed. Since the projected area after reflection is formed by the projection area being reflected by the plane mirror, the position of the projected area after reflection is related to the position of the plane mirror. Therefore, the position of the plane mirror must be the same when capturing the second and third target images. If the cockpit camera simultaneously captures the second and third target images of the projection area after reflection by the plane mirror, the position of the plane mirror does not change, and thus, there is no need to fix the position of the plane mirror.

[0161] S505 , determining the coordinates of the projection area in the cockpit camera coordinate system according to the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the plane reflection imaging principle.

[0162] Since the reflected projection area is formed after the projection area is reflected by the plane mirror, in the cockpit camera coordinate system, the projection area and the reflected projection area still satisfy the plane reflection imaging principle. Therefore, the electronic device can determine the distance from the reflected projection area to the plane mirror in the cockpit camera coordinate system based on the coordinates of the plane mirror in the cockpit camera coordinate system and the coordinates of the reflected projection area in the cockpit camera coordinate system.

[0163] Next, the electronic device can determine the coordinates of the projection area in the cockpit camera coordinate system based on the above distance, the plane reflection principle, and the coordinates of the plane mirror in the cockpit camera coordinate system.

[0164] As an embodiment, the coordinates of the projection area in the cockpit camera coordinate system can be determined by an auxiliary camera. A pre-calibrated auxiliary camera is set up at a position within the shooting range that includes the projection area, and the image of the projection area captured by the auxiliary camera is obtained. Based on the intrinsic parameters of the auxiliary camera, the coordinates of the projection area in the image coordinate system are converted to the coordinates in the auxiliary camera coordinate system. Based on the extrinsic parameters of the auxiliary camera, the coordinates of the projection area in the auxiliary camera coordinate system are converted to the coordinates in the world coordinate system. Furthermore, based on the positional relationship between the cockpit camera and the auxiliary camera in the world coordinate system and the extrinsic parameters of the cockpit camera, the coordinates of the projection area in the world coordinate system are converted to the coordinates of the projection area in the cockpit camera coordinate system.

[0165] As can be seen, in the embodiments of the present application, the electronic device can obtain a second target image captured by the cockpit camera, which is a projection area showing a first target and reflected by a plane mirror; determine the coordinates of the reflected projection area in the cockpit camera coordinate system based on pre-calibrated cockpit camera parameters and the pixel coordinates of the first target in the second target image; obtain a third target image captured by the cockpit camera, which is a plane mirror with a second target attached to its surface; determine the coordinates of the plane mirror in the cockpit camera coordinate system based on pre-calibrated cockpit camera parameters and the pixel coordinates of the second target in the third target image; and determine the coordinates of the projection area in the cockpit camera coordinate system based on the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the principle of plane reflection imaging. Since the reflected projection area is formed by the projection area being reflected by the plane mirror, the projection area in the cockpit camera coordinate system and the reflected projection area also satisfy the principle of plane reflection imaging. Therefore, the electronic device can determine the coordinates of the projection area in the cockpit camera coordinate system based on the principle of plane reflection imaging, the coordinates of the plane mirror in the cockpit camera coordinate system, and the coordinates of the reflected projection area in the cockpit camera coordinate system. The above process only uses the intrinsic parameters of the cockpit camera, without using the extrinsic parameters of the cockpit camera, and there is no need to convert the coordinates to the world coordinate system. The coordinates of the projection area in the cockpit camera coordinate system can be quickly determined.

[0166] As an implementation method of the present application, Figure 6 As shown, the step of obtaining the target sight line data of the user with respect to the calibration mark captured by the cockpit camera may include:

[0167] S601, controlling the projection area of ​​the cockpit to display calibration marks, and recording the number of displayed calibration marks;

[0168] In order to determine whether the number of displayed calibration marks has reached a preset number, the electronic device may record the number of displayed calibration marks while controlling the projection area of ​​the cockpit to display the calibration marks.

[0169] For example, if the electronic device has displayed two calibration marks before displaying the calibration mark in the projection area of ​​the control cabin, the number of displayed calibration marks recorded by the electronic device is 2.

[0170] For another example, if the electronic device has not displayed a calibration mark before displaying the calibration mark in the projection area of ​​the control cabin, the number of displayed calibration marks recorded by the electronic device is 0.

[0171] In the embodiment of the present application, there is no specific limitation on the number and movement trajectory of the calibration marks. For example, the calibration marks can be dots displayed one by one along a circular trajectory, dots at fixed positions, etc.

[0172] S602, obtaining the user's sight line data with respect to the displayed calibration mark captured by the cockpit camera;

[0173] To determine the direction of the user's gaze at the calibration marker, a cockpit camera can be used to capture the user's gaze data relative to the displayed calibration marker. In this way, the electronic device can obtain the gaze data captured by the cockpit camera. For example, the gaze data can include an image of the user gazing at the calibration marker.

[0174] As one implementation of the present application, each time the electronic device controls the projection area to display a calibration mark, the cockpit camera will capture the user's line of sight data regarding that calibration mark. For example, when the electronic device controls the projection area to display a calibration mark, the cockpit camera will capture the user's line of sight data regarding that calibration mark. The next time the electronic device controls the projection area to display a calibration mark, the cockpit camera will also capture the user's line of sight data regarding the next calibration mark.

[0175] S603, determine whether the number of recorded and displayed calibration marks reaches the preset number; if the number of recorded and displayed calibration marks does not reach the preset number, return to step S601; if the number of recorded and displayed calibration marks reaches the preset number, execute step S604.

[0176] To further improve the accuracy of sight line calibration, the electronic device can pre-set the number of calibration marks. Thus, the electronic device can perform sight line calibration for the user based on the preset number of calibration marks. The electronic device can then determine whether the number of recorded displayed calibration marks has reached the preset number.

[0177] For example, if the preset number is 5 and the number of displayed calibration marks is 4, the electronic device will return to execute step S601 and control the projection area of ​​the cockpit to display the 5th calibration mark.

[0178] S604: Determine the acquired sight line data as the user's target sight line data for the calibration mark.

[0179] When the number of recorded displayed calibration marks reaches a preset number, it indicates that sufficient sight line data has been acquired. Then, the electronic device can determine the acquired sight line data as the user's target sight line data for the calibration marks.

[0180] Continuing with the previous example, after the electronic device controls the cockpit projection area to display the fifth calibration mark, the electronic device can obtain the user's line of sight data for the fifth calibration mark, as captured by the cockpit camera. Furthermore, if it determines that the number of displayed calibration marks has reached a preset number, the electronic device can determine the line of sight data corresponding to each of the five displayed calibration marks as the user's target line of sight data for the five displayed calibration marks.

[0181] As can be seen, in an embodiment of the present application, the electronic device can control the projection area of ​​the cockpit to display calibration marks and record the number of displayed calibration marks; obtain the user's line of sight data captured by the cockpit camera with respect to the displayed calibration marks; determine whether the number of recorded displayed calibration marks reaches a preset number; if the number of recorded displayed calibration marks does not reach the preset number, return to the step of controlling the projection area of ​​the cockpit to display the calibration marks and record the number of displayed calibration marks; if the number of recorded displayed calibration marks reaches the preset number, determine the obtained line of sight data as the user's target line of sight data with respect to the calibration marks. Because the electronic device can obtain line of sight data corresponding to a preset number of calibration marks and use the line of sight data corresponding to the preset number of calibration marks as target line of sight data, it can subsequently perform line of sight calibration on the user based on multiple sets of target line of sight data. In this way, the accuracy of line of sight calibration can be further improved.

[0182] As an implementation manner of an embodiment of the present application, before the step of determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight data, the method may further include:

[0183] The target sight line data is cleaned to obtain cleaned target sight line data.

[0184] Because the user's gaze typically moves from somewhere to the calibration mark while they are gazing at it, the target gaze data may include data from the user's gaze movement. Furthermore, the user may be distracted, meaning their gaze may be located far from the calibration mark. Therefore, to determine the target gaze data corresponding to the user's gaze at the calibration mark, the electronic device may clean the target gaze data to obtain the cleaned target gaze data.

[0185] In one embodiment, the electronic device may cleanse a set of target sight line data after acquiring the set. For example, if there are two sets of target sight line data, the electronic device may cleanse the first set of target sight line data after acquiring the first set of target sight line data. After acquiring the second set of target sight line data, the electronic device may cleanse the second set of target sight line data.

[0186] In another embodiment, the electronic device may first obtain a preset number of sets of target sight data, and then perform data cleaning on each set of target sight data. For example, if there are two sets of target sight data, the electronic device may obtain a first set of target sight data and a second set of target sight data, and then perform data cleaning on the first set of target sight data and the second set of target sight data.

[0187] The step of determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight data may include:

[0188] Based on the cleaned target sight line data, the coordinates of the user's target sight line landing point for the calibration mark are determined in the cockpit camera coordinate system.

[0189] Since the cleaned target sight data can more accurately reflect the position where the user is looking at the calibration mark, after obtaining the cleaned target sight data, the electronic device can determine the coordinates of the user's target sight landing point for the calibration mark in the cockpit camera coordinate system based on the cleaned target sight data.

[0190] As can be seen, in this embodiment of the present application, the electronic device can clean the target sight data to obtain cleaned target sight data; based on the cleaned target sight data, the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system are determined. Because the target sight data may contain instances where the user's sight line does not fall on the calibration mark, the electronic device can clean the target sight data. This cleaned target sight data can more accurately reflect the location where the user is looking at the calibration mark, further improving the accuracy of sight calibration.

[0191] As an implementation of an embodiment of the present application, the aforementioned sight line data may include multiple sight line position coordinates of the user captured by the cockpit camera within a preset time period after the calibration mark is displayed. In this case, the aforementioned step of cleaning the target sight line data to obtain cleaned target sight line data may include performing data cleaning to obtain cleaned target sight line data using one of the following methods:

[0192] Clustering the plurality of sight line position coordinates to determine a cluster center; and determining the cluster center as the cleaned target sight line data;

[0193] Clustering the plurality of sight line position coordinates to determine a category including the largest number of sight line position coordinates; and determining the sight line position coordinates included in the determined category as cleaned target sight line data;

[0194] Determining a user's sight focus set based on a capture time sequence corresponding to the plurality of sight position coordinates; determining an end point position coordinate of the sight focus set as cleaned target sight data;

[0195] Based on the capture timing corresponding to the multiple line of sight position coordinates, the user's line of sight focus set is determined; the line of sight position coordinates whose distance from the end position coordinates of the line of sight focus set is less than a preset threshold are determined as the cleaned target line of sight data.

[0196] In the first embodiment, since the distance between the corresponding line of sight position coordinates when the user is distracted is far from most of the other line of sight position coordinates, and the time of the sliding process when the user's line of sight slides to the calibration mark is much shorter than the time when the line of sight finally stays on the calibration mark, the number of line of sight position coordinates corresponding to the sliding process is much smaller than the number of line of sight position coordinates corresponding to when the line of sight finally stays on the calibration mark.

[0197] Based on the above reasons, the electronic device can cluster the multiple sight line position coordinates obtained within a preset time period after displaying the calibration mark, determine the cluster center corresponding to the multiple sight line position coordinates, and then determine the cluster center as the cleaned target sight line data.

[0198] For example, in the cockpit camera coordinate system, multiple sight line coordinates are (10, 21, 5), (13, 21, 5), and (7, 21, 5). The electronic device can cluster the multiple sight line coordinates and determine the cluster center as (10, 21, 5). Then, the cluster center (10, 21, 5) is determined as the cleaned target sight line data.

[0199] As an implementation method, the electronic device can cluster multiple sight line position coordinates obtained within a preset time period after displaying the calibration mark, determine the cluster center corresponding to the multiple sight line position coordinates, and determine the sight line position coordinates whose distance from the cluster center is less than a preset threshold as the cleaned target sight line data.

[0200] For example, in the cockpit camera coordinate system, the multiple sight line position coordinates are (24, 15, 32), (25, 15, 32), (23, 15, 32), (10, 21, 5), (13, 21, 5), and (7, 21, 5). The electronic device can cluster the multiple sight line position coordinates and determine the cluster center as (24, 15, 32). If the preset threshold is 2, the electronic device can determine that the sight line position coordinates whose distance from the cluster center is less than the preset threshold are (23, 15, 32) and (25, 15, 32). (23, 15, 32), (25, 15, 32), and (24, 15, 32) are determined as the cleaned target sight line data.

[0201] In the second embodiment, because the cockpit camera continuously collects the user's gaze position coordinates while the calibration mark is displayed, after the user's gaze stabilizes on the calibration mark, a large number of gaze position coordinates are concentrated near the calibration mark. The electronic device can cluster the multiple gaze position coordinates to obtain multiple categories. The category containing the largest number of gaze position coordinates is determined, and the gaze position coordinates included in this category are determined as the cleaned target gaze data.

[0202] For example, after clustering multiple gaze position coordinates, the electronic device obtains three categories: category 1, category 2, and category 3. Category 1 includes the gaze position coordinates (7, 10, 32), (8, 10, 31), and (7, 11, 30); category 2 includes the gaze position coordinates (15, 16, 31), (16, 15, 31), and (14, 17, 30); category 3 includes the gaze position coordinates (26, 16, 31), (25, 15, 31), (24, 15, 32), (25, 15, 32), and (25, 16, 30). Since category 3 includes the largest number of gaze position coordinates, the electronic device can use the gaze position coordinates included in category 3 as the cleaned target gaze data.

[0203] In a third embodiment, as the user gazes at the calibration mark, their gaze position coordinates start moving from a certain location and eventually stop at the calibration mark. Therefore, to cleanse the user's gaze position coordinates as they move, the electronic device can determine a set of the user's gaze focal points based on the capture time sequence corresponding to the multiple gaze position coordinates. The gaze focal point set is a sequence of the multiple gaze focal points arranged in the order of their acquisition time.

[0204] Since the end point of the user's gaze focus set is the coordinates of the user's gaze position at the calibration mark, the electronic device can determine the coordinates of the end point of the gaze focus set as the cleaned target gaze data. In other words, data cleaning is performed using a data cleaning method based on time series analysis.

[0205] For example, in the cockpit camera coordinate system, the starting position coordinates and the end position coordinates of the user's line of sight focus set determined by the electronic device are (10, 15, 21) and (16, 25, 31), respectively. Then the electronic device can determine the end position coordinates (16, 25, 31) as the cleaned target line of sight data.

[0206] In a fourth embodiment, because the endpoint of the gaze focus set corresponds to the coordinates of the user's gaze position when they gazed at the calibration mark, the electronic device can determine the user's gaze focus set based on the capture time sequence corresponding to the multiple gaze position coordinates. Furthermore, the gaze position coordinates whose distance from the endpoint of the gaze focus set is less than a preset threshold are determined as the cleaned target gaze data.

[0207] For example, the electronic device acquires the following coordinates for the sight focus positions: (12, 13, 20), (14, 15, 21), (24, 15, 32), (25, 15, 32), and (25, 16, 30), where the endpoint of the sight focus set is (25, 16, 30). If the preset threshold is 2, the electronic device can determine that the sight position coordinates whose distance from the endpoint of the sight focus set is less than the preset threshold are (24, 15, 32) and (25, 15, 32). Therefore, the electronic device can use (24, 15, 32), (25, 15, 32), and (25, 16, 30) as the cleaned target sight data.

[0208] As an implementation, the electronic device may perform data cleaning by removing outliers. For example, the electronic device may calculate the distance between each of the multiple collected sight line position coordinates and other sight line position coordinates. If a distance greater than a preset distance exists among the smallest preset number of such distances, the sight line position coordinate is determined to be an outlier and is deleted.

[0209] As can be seen, in an embodiment of the present application, an electronic device can cluster multiple gaze position coordinates to determine a cluster center; determine the cluster center as the cleaned target gaze data; cluster the multiple gaze position coordinates to determine the category containing the largest number of gaze position coordinates; determine the gaze position coordinates included in the determined category as the cleaned target gaze data; determine a user's gaze focus set based on the capture time sequence corresponding to the multiple gaze position coordinates; determine the end position coordinates of the gaze focus set as the cleaned target gaze data; determine the user's gaze focus set based on the capture time sequence corresponding to the multiple gaze position coordinates; and determine the gaze position coordinates whose distance from the end position coordinates of the gaze focus set is less than a preset threshold as the cleaned target gaze data. Because the gaze position coordinates corresponding to a distracted user are farther away from most other gaze position coordinates, and the number of gaze position coordinates corresponding to a user's gaze sliding process is much smaller than the number of gaze position coordinates corresponding to a final gaze stop on a calibration mark, the electronic device can perform data cleaning using clustering. Since the user's gaze at the calibration mark will eventually stop at the calibration mark, the end position coordinates of the gaze focus set are determined as the cleaned target gaze data. Since the user's line of sight is stabilized on the calibration mark, a large number of line of sight position coordinates are concentrated near the calibration mark, so the category obtained by clustering and including the largest number of line of sight position coordinates is determined as the target line of sight data after cleaning. Since the end position of the line of sight focus set corresponds to the line of sight position coordinates when the user is looking at the calibration mark, the line of sight position coordinates whose distance from the end position coordinates of the line of sight focus set is less than a preset threshold are determined as the target line of sight data after cleaning. In this way, the target line of sight data after cleaning can more accurately reflect the position where the user is looking at the calibration mark, and the accuracy of the line of sight calibration can be further improved. In the case where there are multiple line of sight position coordinates after cleaning, since there are multiple target line of sight position coordinates used for line of sight calibration, the accuracy of the line of sight calibration can be improved.

[0210] As an implementation of an embodiment of the present application, the step of determining the sight line calibration parameters based on the difference between the target sight line landing point coordinates and the predicted sight line landing point coordinates may include:

[0211] Based on the target sight point coordinates and the predicted sight point coordinates, the sight calibration matrix R is calculated according to the following formula:

[0212] R 3×3 =(V′ 3×n V′ n×3 T ) -1 V′ 3×n V n×3 T )T

[0213] Where n is the number of target sight point coordinates, V′ 3×n is a matrix whose column vectors are composed of the components of the predicted sight point coordinates, V′ n×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors consist of the components of the coordinates of the target sight point.

[0214] Since the target and predicted sight point coordinates are for the same calibration marker, they have the same number of components. Since both are in the cockpit camera coordinate system, they both contain three components.

[0215] If v i represents the coordinates of the user's target sight point for the i-th calibration mark, v i ′ represents the predicted sight point coordinates for the i-th calibration mark. The predicted sight point coordinates can be constructed. The number of target sight point coordinates and predicted sight point coordinates is n, that is, the value of i is 1, 2…n, then the n target sight point coordinates and n predicted sight point coordinates are represented by matrices, which are [v1, v2,…, v n ] 3×n and [v1′v2′,…,v n ′] 3×n .

[0216] If the sight calibration parameter is the sight calibration matrix R, then the electronic device can establish the relationship between the matrix corresponding to the target sight point coordinates and the predicted sight point coordinates as [v1,v2,…,v n ] 3×n =R 3×3 [v1′v2′,…,v n ′] 3×n This formula is an overdetermined equation. Electronic devices can use the least squares method to solve the overdetermined equation. That is, electronic devices can solve the overdetermined equation according to the formula R 3×3 =((V′ 3×n V′ n×3 T ) -1 V′ 3×n V n×3 T ) T , solve the sight calibration matrix R.

[0217] It can be seen that in the embodiment of the present application, the electronic device can calculate the sight calibration matrix R based on the target sight point coordinates and the predicted sight point coordinates according to the following formula: R 3×3 =(V′ 3×n V′ n×3 T ) -1 V′ 3×n V n×3 T ) T Where n is the number of the target sight point coordinates, V′ 3×n is a matrix whose column vectors are composed of the components of the predicted sight point coordinates, V′ n×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors are composed of the components of the target sight point coordinates. Since the target sight point coordinates and the predicted sight point coordinates are for the same calibration mark, the number of target sight point coordinates and the predicted sight point coordinates is the same. Since the target sight point coordinates and the predicted sight point coordinates are both in the cockpit camera coordinate system, the target sight point coordinates and the predicted sight point coordinates both contain three components. The matrices corresponding to the target sight point coordinates and the predicted sight point coordinates can be expressed as [v1,v2,…,v n ] 3×n and [v1′v2′,…,v n ′] 3×n . Electronic devices can construct the formula [v1,v2,…,v n ] 3×n =R 3×3 [v1′v2′,…,v n ′] 3×n , and then the least square method is used to solve the sight calibration matrix R. In this way, the solved sight calibration matrix can be made more accurate.

[0218] As an implementation method of the present application, Figure 7 As shown, after the step of determining the sight line calibration parameters based on the difference between the target sight line landing point coordinates and the predicted sight line landing point coordinates, the method may further include:

[0219] S701, obtaining the user's current sight line data;

[0220] After determining the sight calibration matrix, in order to calibrate the user's sight, the electronic device can obtain the user's current sight data, where the current sight data can be the user's current sight coordinates in the cockpit camera coordinate system.

[0221] S702: Determine the product of the sight line calibration matrix and the current sight line data as calibrated sight line data.

[0222] After obtaining the user's current sight data, the electronic device can use the formula v=Rv ′ , the sight calibration matrix R and the current sight data v ′ Multiply them together to get the calibrated sight data v.

[0223] As an implementation, the electronic device can determine, based on the user's calibrated line of sight data, whether the user's line of sight has been permanently away from the front of the cabin, and thus determine whether the user is experiencing driving fatigue, distraction, or other such conditions. If the user is in such a state, the electronic device can control the cabin audio output device to play an audio alert to the driver, thereby reducing the occurrence of traffic accidents.

[0224] As another implementation, the electronic device can determine whether the user's gaze point is within a wake-up zone based on the user's calibrated line of sight data. If the user's gaze point is within a wake-up zone, the electronic device can wake up a device within the cabin corresponding to that wake-up zone. These devices may include at least one of an in-vehicle display, audio system, air conditioning, windshield wipers, and an electric heater, though these are not specifically limited here.

[0225] In one embodiment, the sight calibration matrix is ​​a 3×3 matrix, and the current sight data can be represented as a three-dimensional column vector. The electronic device can multiply the sight calibration matrix with the current sight data, and the resulting product is also a three-dimensional column vector, which is the calibrated sight data.

[0226] As can be seen, in the embodiments of the present application, the electronic device can obtain the user's current line of sight data; the product of the line of sight calibration matrix and the current line of sight data is determined as the calibrated line of sight data. In order to calibrate the user's line of sight, the electronic device can obtain the user's current line of sight data. After obtaining the user's current line of sight data, the electronic device can multiply the line of sight calibration matrix and the current line of sight data, and use the product as the calibrated line of sight data. In this way, line of sight calibration can be achieved with high accuracy.

[0227] As an implementation of an embodiment of the present application, before the step of obtaining the target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method may further include:

[0228] Controlling the cockpit's projection area to display sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area, and / or,

[0229] The cockpit audio output device is controlled to play a sight line calibration prompt voice, wherein the sight line calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area.

[0230] In the first embodiment, to guide the user to perform sight calibration, that is, to look at the calibration mark in the projection area, the electronic device can control the projection area of ​​the cockpit to display a sight calibration prompt. For example, the sight calibration prompt can be a text message such as "Please look at the calibration mark".

[0231] In a second embodiment, to guide the user to perform sight calibration, namely, to focus on the calibration mark in the projection area, the electronic device can control the cabin audio output device to play a sight calibration prompt. For example, the sight calibration prompt could read, "Please focus on the calibration mark displayed in the projection area ahead."

[0232] It can be seen that in the embodiment of the present application, the electronic device can control the projection area of ​​the cockpit to display sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area, and / or control the cockpit audio output device to play the sight calibration prompt voice, wherein the sight calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area. In order to guide the user to perform sight calibration and then obtain the user's target sight data, the electronic device can control the projection area of ​​the cockpit to display sight calibration prompt information, and / or control the cockpit audio output device to play the sight calibration prompt voice. In this way, the user can be guided so that the user looks at the calibration mark in the projection area and then achieves sight calibration.

[0233] As an implementation of an embodiment of the present application, the calibration mark is displayed in a projection area of ​​a head-up display HUD in the cockpit.

[0234] Because the HUD is an integral part of the cockpit, no additional equipment is required. The location of the HUD's corresponding projection area is known, allowing the precise positioning of the calibration marker within the projection area. Furthermore, the HUD can display any number of calibration markers in any position within the projection area, enhancing the flexibility of calibration marker display. Furthermore, using the HUD for projection clearly projects the calibration markers, making them easier for the user to view. Therefore, using the HUD to project calibration markers enables in-cabin line-of-sight calibration with high accuracy.

[0235] As an implementation of the embodiment of the present application, the flowchart of the method for determining the sight calibration parameters can be as follows: Figure 8 Specifically, the following steps may be included:

[0236] S801, guiding the user to start calibration;

[0237] To encourage the user to focus on the calibration mark displayed in the projection area, the electronic device can guide the user to begin calibration. Specifically, the electronic device can guide the user to begin gaze calibration by controlling the projection area to display gaze calibration prompts, controlling the cabin audio output device to play gaze calibration prompts, and displaying a pattern in the projection area to guide the user to begin gaze calibration. It should be noted that the embodiments of this application do not specifically limit the method for guiding the user to begin calibration.

[0238] The calibration mark number i can be used to indicate the number of calibration marks currently displayed. When the sight calibration starts, the initial calibration mark number i is 0.

[0239] S802, controlling the projection area to display the i-th calibration mark, while guiding the user to look at the calibration mark displayed in the projection area;

[0240] In order to enable the cockpit camera to capture the user's target sight data of the calibration mark, the electronic device can control the projection area to display the i-th calibration mark, and at the same time guide the user to look at the calibration mark displayed in the projection area.

[0241] S803, synchronous data collection;

[0242] While the user is gazing at the calibration mark, the cockpit camera can capture the user's target sight line data for the calibration mark, and then the electronic device can collect the target sight line data captured by the cockpit camera.

[0243] S804, data cleaning;

[0244] After the electronic device collects the user's target sight line data for the i-th calibration mark, the electronic device may clean the collected target sight line data so that the cleaned target sight line data more accurately represents the position where the user is looking at the calibration mark.

[0245] S805, determine whether the calibration identification number i is greater than or equal to the preset number N; if not, return to step S802, and set i=i+1; if yes, execute step S806.

[0246] In order to cyclically collect N groups of target sight line data corresponding to N calibration marks, the electronic device can determine whether the calibration mark number i is greater than or equal to a preset number N.

[0247] S806: Solve and output sight line calibration parameters.

[0248] The predicted sight point coordinates are the estimated sight position of the user looking at a calibration mark. The electronic device can determine and output sight calibration parameters based on the target sight point coordinates and the predicted sight point coordinates.

[0249] It can be seen that in the embodiment of the present application, the electronic device can guide the user to start calibration; control the projection area to display the i-th calibration mark, and at the same time guide the user to look at the calibration mark displayed in the projection area; synchronously collect data; data cleaning; determine whether the calibration mark number i is greater than or equal to the preset number N; if not, display the next calibration mark; if so, solve and output the line of sight calibration parameters. Since the electronic device can obtain the line of sight data corresponding to a preset number of calibration marks, and use the line of sight data corresponding to the preset number of calibration marks as target line of sight data, it is possible to perform line of sight calibration on the user based on multiple sets of target line of sight data. In this way, line of sight calibration can be achieved with high accuracy.

[0250] It should be noted that in the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0251] Corresponding to the above-mentioned line of sight calibration method, an embodiment of the present application further provides a line of sight calibration device. The line of sight calibration device provided in the embodiment of the present application is introduced below.

[0252] like Figure 9 As shown, a sight line calibration device, the device comprising:

[0253] The sight line data acquisition module 901 is used to acquire the target sight line data of the user toward the calibration mark captured by the cockpit camera, wherein the calibration mark is displayed in the projection area of ​​the cockpit and indicates the position where the user is looking at the calibration mark. The projection area is the area in front of the cockpit for displaying projection information.

[0254] A predicted sight line coordinate determination module 902 is configured to determine the coordinates of the calibration marker in the cockpit camera coordinate system as the predicted sight line landing point coordinates based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration marker in the projection area;

[0255] A target sight line coordinate determination module 903 is configured to determine the coordinates of the user's target sight line landing point for the calibration mark in the cockpit camera coordinate system based on the target sight line data;

[0256] The sight line calibration parameter determination module 904 is configured to determine the sight line calibration parameters according to the target sight line landing point coordinates and the predicted sight line landing point coordinates.

[0257] As can be seen, in the solution provided by the embodiment of the present application, the electronic device can obtain the target sight line data of the user toward the calibration mark captured by the cabin camera, wherein the calibration mark is displayed in the projection area of ​​the cabin and is used to indicate the location of the calibration mark at which the user is looking. The projection area is the area located in front of the cabin for displaying projected information. Based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration mark in the projection area, the coordinates of the calibration mark in the cabin camera coordinate system are determined as the predicted sight line landing point coordinates. Based on the target sight line data, the coordinates of the user's target sight line landing point toward the calibration mark in the cabin camera coordinate system are determined. The sight line calibration parameters are determined based on the target sight line landing point coordinates and the predicted sight line landing point coordinates. Since the electronic device can control the projection area to display the calibration mark, and the coordinates of the calibration mark in the projection area are known, the electronic device can accurately determine the coordinates of the calibration mark in the cabin camera coordinate system, i.e., the predicted sight line landing point coordinates, based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration mark in the projection area. In this way, the electronic device can accurately determine the calibration parameters according to the difference between the predicted sight point coordinates and the target sight point coordinates, thereby realizing sight calibration with high accuracy.

[0258] As an implementation of the embodiment of the present application, the above-mentioned device may further include:

[0259] A first target image acquisition module is used to acquire a first target image of the projection area captured by the cockpit camera after being reflected by the plane mirror;

[0260] a first projection area coordinate determination module, configured to determine the coordinates of the projection area after reflection in a world coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the projection area in the first target image;

[0261] a second projection area coordinate determination module, configured to determine the coordinates of the projection area in the world coordinate system according to the distance between the projection area and the plane mirror, the coordinates of the reflected projection area in the world coordinate system, and the plane reflection imaging principle;

[0262] The third projection area coordinate determination module is used to convert the coordinates of the projection area in the world coordinate system into the coordinates of the projection area in the cockpit camera coordinate system according to a pre-calibrated conversion relationship between the world coordinate system and the cockpit camera coordinate system.

[0263] As an implementation of the embodiment of the present application, the above-mentioned device may further include:

[0264] A second target image acquisition module is used to acquire a second target image of the projection area showing the first target captured by the cockpit camera after being reflected by the plane mirror;

[0265] a fourth projection area coordinate determination module, configured to determine the coordinates of the reflected projection area in the cockpit camera coordinate system based on pre-calibrated cockpit camera parameters and the pixel coordinates of the first target in the second target image;

[0266] A third target image acquisition module is used to acquire a third target image of the plane mirror with the second target attached to the surface captured by the cockpit camera;

[0267] a plane mirror coordinate determination module, configured to determine the coordinates of the plane mirror in the cockpit camera coordinate system based on pre-calibrated parameters of the cockpit camera and the pixel coordinates of the second target in the third target image;

[0268] a fifth projection area coordinate determination module, configured to determine the coordinates of the projection area in the cockpit camera coordinate system according to the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the plane reflection imaging principle.

[0269] As an implementation of an embodiment of the present application, the target sight line coordinate determination module 903 may include:

[0270] A first calibration mark display submodule, configured to control the projection area of ​​the cockpit to display the calibration mark and record the number of displayed calibration marks;

[0271] a sight line data acquisition submodule, configured to acquire sight line data of the user with respect to the displayed calibration mark captured by the cockpit camera;

[0272] The quantity determination submodule is used to determine whether the number of recorded and displayed calibration marks reaches a preset number;

[0273] a second calibration mark display submodule, configured to return to the projection area of ​​the control cabin to display the calibration mark and record the number of displayed calibration marks if the number of displayed calibration marks does not reach the preset number;

[0274] The target sight line data determination submodule is configured to determine the acquired sight line data as the user's target sight line data for the calibration marks if the number of the recorded displayed calibration marks reaches the preset number.

[0275] As an implementation of the embodiment of the present application, the above-mentioned device may further include:

[0276] A data cleaning module, configured to clean the target sight line data to obtain cleaned target sight line data;

[0277] The target sight line coordinate determination module 903 may include:

[0278] The target sight point coordinate submodule is used to determine the target sight point coordinates of the user for the calibration mark in the cockpit camera coordinate system based on the cleaned target sight data.

[0279] As an implementation of an embodiment of the present application, the above-mentioned sight line data may include multiple sight line position coordinates of the user captured by the cabin camera within a preset time period after the calibration mark is displayed;

[0280] The data cleaning module may include:

[0281] A first cleaning submodule is configured to cluster the plurality of sight line position coordinates to determine a cluster center; and determine the cluster center as the cleaned target sight line data;

[0282] A second cleaning submodule is configured to cluster the plurality of sight line position coordinates to determine a category including the largest number of sight line position coordinates; and determine the sight line position coordinates included in the determined category as cleaned target sight line data;

[0283] The third cleaning submodule is used to determine the user's sight focus set based on the capture time sequence corresponding to the multiple sight position coordinates; and determine the end position coordinates of the sight focus set as the cleaned target sight data.

[0284] The fourth cleaning submodule is used to determine the user's line of sight focus set based on the capture timing corresponding to the multiple line of sight position coordinates; and determine the line of sight position coordinates whose distance from the end position coordinates of the line of sight focus set is less than a preset threshold as the cleaned target line of sight data.

[0285] As an implementation of an embodiment of the present application, the sight calibration parameter determination module 904 may include:

[0286] The calibration matrix calculation submodule is used to calculate the sight calibration matrix R based on the target sight point coordinates and the predicted sight point coordinates according to the following formula:

[0287] R 3×3 =((V′ 3×n V′ n×3 T ) -1 V′ 3×n V n×3 T )T

[0288] Where n is the number of target sight point coordinates, V′ 3×n is a matrix whose column vectors are composed of the components of the predicted sight point coordinates, V′ n×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors consist of the components of the coordinates of the target sight point.

[0289] As an implementation of the embodiment of the present application, the above-mentioned device may further include:

[0290] A current sight line data acquisition module, configured to acquire the current sight line data of the user;

[0291] The sight line data calibration module is configured to determine the product of the sight line calibration matrix and the current sight line data as the calibrated sight line data.

[0292] As an implementation of the embodiment of the present application, the above-mentioned device may further include:

[0293] A sight calibration prompt information display module, used to control the projection area of ​​the cockpit to display a sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area;

[0294] The sight calibration prompt voice playing module is used to control the cockpit audio output device to play the sight calibration prompt voice, wherein the sight calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area.

[0295] As an implementation of an embodiment of the present application, the calibration mark is displayed in a projection area of ​​a head-up display (HUD) in the cockpit.

[0296] The present application also provides an electronic device, such as Figure 10 Shown, including:

[0297] Memory 1001, used for storing computer programs;

[0298] The processor 1002 is configured to implement the sight line calibration method steps described in any of the above embodiments when executing the program stored in the memory 1001.

[0299] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1002, the communication interface, and the memory 1001 communicate with each other via the communication bus.

[0300] As can be seen, in the solution provided by the embodiment of the present application, the electronic device can obtain the target sight line data of the user toward the calibration mark captured by the cabin camera, wherein the calibration mark is displayed in the projection area of ​​the cabin and is used to indicate the location of the calibration mark at which the user is looking. The projection area is the area located in front of the cabin for displaying projected information. Based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration mark in the projection area, the coordinates of the calibration mark in the cabin camera coordinate system are determined as the predicted sight line landing point coordinates. Based on the target sight line data, the coordinates of the user's target sight line landing point toward the calibration mark in the cabin camera coordinate system are determined. The sight line calibration parameters are determined based on the target sight line landing point coordinates and the predicted sight line landing point coordinates. Since the electronic device can control the projection area to display the calibration mark, and the coordinates of the calibration mark in the projection area are known, the electronic device can accurately determine the coordinates of the calibration mark in the cabin camera coordinate system, i.e., the predicted sight line landing point coordinates, based on the pre-calibrated coordinates of the projection area in the cabin camera coordinate system and the coordinates of the calibration mark in the projection area. In this way, the electronic device can accurately determine the calibration parameters according to the difference between the predicted sight point coordinates and the target sight point coordinates, thereby realizing sight calibration with high accuracy.

[0301] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0302] The communication interface is used for communication between the above electronic device and other devices.

[0303] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0304] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0305] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above-mentioned line of sight calibration methods are implemented.

[0306] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the sight calibration methods in the above embodiments.

[0307] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0308] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0309] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between other embodiments. In particular, the device, electronic device, computer-readable storage medium, and computer program product embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0310] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A sight line calibration method, characterized in that: The method comprises: Acquiring target sight data of a user toward a calibration marker captured by a cockpit camera, wherein the calibration marker is displayed in a projection area of ​​the cockpit, the calibration marker being used to indicate a location at which the user is looking at the calibration marker, and the projection area being an area in front of the cockpit for displaying projection information; Based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration mark in the projection area, determining the coordinates of the calibration mark in the cockpit camera coordinate system as the predicted sight point coordinates; Determining the target sight point coordinates of the user with respect to the calibration mark in the cockpit camera coordinate system based on the target sight line data; Based on the target sight point coordinates and the predicted sight point coordinates, the sight calibration matrix R is calculated according to the following formula: R 3×3 =((V′ 3×n V′ n×3 T ) -1 V′ 3×n V n×3 T ) T Where n is the number of target sight point coordinates, V′ 3×n is a matrix whose column vectors are composed of the components of the predicted sight point coordinates, V′ 3×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors consist of the components of the coordinates of the target sight point.

2. The method according to claim 1, characterized in that Before the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method further includes: Acquire a first target image of the projection area captured by the cockpit camera and reflected by a plane mirror; Determining coordinates of the reflected projection area in a world coordinate system based on pre-calibrated parameters of the cockpit camera and pixel coordinates of the projection area in the first target image; Determine the coordinates of the projection area in the world coordinate system according to the distance between the projection area and the plane mirror, the coordinates of the reflected projection area in the world coordinate system, and the plane reflection imaging principle; According to a pre-calibrated conversion relationship between the world coordinate system and the cockpit camera coordinate system, the coordinates of the projection area in the world coordinate system are converted into the coordinates of the projection area in the cockpit camera coordinate system.

3. The method according to claim 1, characterized in that Before the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method further includes: Acquire a second target image of the projection area showing the first target captured by the cockpit camera and reflected by the plane mirror; determining the coordinates of the reflected projection area in the cockpit camera coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the first target in the second target image; Acquiring a third target image of the plane mirror with a second target attached to the surface thereof, captured by the cockpit camera; Determining the coordinates of the plane mirror in the cockpit camera coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the second target in the third target image; The coordinates of the projection area in the cockpit camera coordinate system are determined according to the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the plane reflection imaging principle.

4. The method according to claim 1, wherein The step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera comprises: Control the projection area of ​​the cockpit to display calibration marks and record the number of calibration marks displayed; Acquiring user's sight line data with respect to the displayed calibration mark captured by the cockpit camera; determining whether the number of recorded displayed calibration marks reaches a preset number; If the recorded number of displayed calibration marks does not reach the preset number, returning to the projection area of ​​the control cabin to display the calibration marks and recording the number of displayed calibration marks; If the number of the recorded displayed calibration marks reaches the preset number, the acquired sight line data is determined as the user's target sight line data for the calibration marks.

5. The method according to claim 4, characterized in that Before the step of determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight data, the method further includes: performing data cleaning on the target sight line data to obtain cleaned target sight line data; The step of determining the coordinates of the user's target sight point with respect to the calibration mark in the cockpit camera coordinate system based on the target sight data includes: Based on the cleaned target sight line data, the coordinates of the user's target sight line landing point for the calibration mark are determined in the cockpit camera coordinate system.

6. The method according to claim 5, characterized in that The sight line data includes a plurality of sight line position coordinates of the user captured by the cockpit camera within a preset time period after the calibration mark is displayed; The step of performing data cleaning on the target sight line data to obtain cleaned target sight line data includes performing data cleaning in one of the following ways to obtain cleaned target sight line data: Clustering the plurality of sight line position coordinates to determine a cluster center; and determining the cluster center as the cleaned target sight line data; Clustering the plurality of sight line position coordinates to determine a category including the largest number of sight line position coordinates; and determining the sight line position coordinates included in the determined category as cleaned target sight line data; Determining a user's sight focus set based on a capture time sequence corresponding to the plurality of sight position coordinates; determining an end point position coordinate of the sight focus set as cleaned target sight data; Determining a gaze focus set of the user based on a capture time sequence corresponding to the multiple gaze position coordinates; The sight line position coordinates whose distance from the end point position coordinates of the sight line focus set is less than a preset threshold are determined as the cleaned target sight line data.

7. The method according to claim 1, characterized in that After the step of calculating the sight line calibration matrix R based on the target sight line landing point coordinates and the predicted sight line landing point coordinates according to the following formula, the method further includes: Obtaining the user's current sight line data; The product of the sight line calibration matrix and the current sight line data is determined as the calibrated sight line data.

8. The method according to any one of claims 1 to 6, characterized in that Before the step of obtaining target sight line data of the user with respect to the calibration mark captured by the cockpit camera, the method further includes: Controlling the cockpit's projection area to display sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area, and / or, The cockpit audio output device is controlled to play a sight line calibration prompt voice, wherein the sight line calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area.

9. The method according to any one of claims 1 to 6, characterized in that The calibration mark is displayed in a projection area of ​​a head-up display (HUD) in the cockpit.

10. A sight line calibration device, characterized in that: The device comprises: a sight line data acquisition module, configured to acquire target sight line data of a user directed at a calibration marker, as captured by a cockpit camera; wherein the calibration marker is displayed in a projection area of ​​the cockpit and indicates the location of the calibration marker to be gazed at by the user; and the projection area is an area in front of the cockpit for displaying projection information; a predicted sight line coordinate determination module, configured to determine the coordinates of the calibration marker in the cockpit camera coordinate system as the predicted sight line landing point coordinates based on the pre-calibrated coordinates of the projection area in the cockpit camera coordinate system and the coordinates of the calibration marker in the projection area; a target sight line coordinate determination module, configured to determine the coordinates of the user's target sight line landing point for the calibration mark in the cockpit camera coordinate system based on the target sight line data; A sight line calibration parameter determination module, configured to determine a sight line calibration parameter according to the target sight line landing point coordinates and the predicted sight line landing point coordinates; The sight line calibration parameter determination module includes: The calibration matrix calculation submodule is used to calculate the sight calibration matrix R based on the target sight point coordinates and the predicted sight point coordinates according to the following formula: R 3×3 =((V′ 3×n V′ n×3 T ) -1 V′ 3×n V n×3 T ) T Where n is the number of target sight point coordinates, V′ 3×n is a matrix whose column vectors are composed of the components of the predicted sight point coordinates, V′ n×3 is a matrix whose row vectors consist of the components of the predicted sight point coordinates, V n×3 is a matrix whose row vectors consist of the components of the coordinates of the target sight point.

11. The device according to claim 10, characterized in that The device further comprises: A first target image acquisition module is used to acquire a first target image of the projection area captured by the cockpit camera after being reflected by the plane mirror; a first projection area coordinate determination module, configured to determine the coordinates of the projection area after reflection in a world coordinate system according to pre-calibrated parameters of the cockpit camera and the pixel coordinates of the projection area in the first target image; a second projection area coordinate determination module, configured to determine the coordinates of the projection area in the world coordinate system according to the distance between the projection area and the plane mirror, the coordinates of the reflected projection area in the world coordinate system, and the plane reflection imaging principle; a third projection area coordinate determination module, configured to convert the coordinates of the projection area in the world coordinate system into the coordinates of the projection area in the cockpit camera coordinate system according to a pre-calibrated conversion relationship between the world coordinate system and the cockpit camera coordinate system; The device further comprises: A second target image acquisition module is used to acquire a second target image of the projection area showing the first target captured by the cockpit camera after being reflected by the plane mirror; a fourth projection area coordinate determination module, configured to determine the coordinates of the reflected projection area in the cockpit camera coordinate system based on pre-calibrated cockpit camera parameters and the pixel coordinates of the first target in the second target image; A third target image acquisition module is used to acquire a third target image of the plane mirror with the second target attached to the surface captured by the cockpit camera; a plane mirror coordinate determination module, configured to determine the coordinates of the plane mirror in the cockpit camera coordinate system based on pre-calibrated parameters of the cockpit camera and the pixel coordinates of the second target in the third target image; a fifth projection area coordinate determination module, configured to determine the coordinates of the projection area in the cockpit camera coordinate system based on the coordinates of the plane mirror in the cockpit camera coordinate system, the coordinates of the reflected projection area in the cockpit camera coordinate system, and the plane reflection imaging principle; The target sight line coordinate determination module includes: A first calibration mark display submodule, configured to control the projection area of ​​the cockpit to display the calibration mark and record the number of displayed calibration marks; a sight line data acquisition submodule, configured to acquire sight line data of the user with respect to the displayed calibration mark captured by the cockpit camera; The quantity determination submodule is used to determine whether the number of recorded and displayed calibration marks reaches a preset number; a second calibration mark display submodule, configured to return to the projection area of ​​the control cabin to display the calibration mark and record the number of displayed calibration marks if the number of displayed calibration marks does not reach the preset number; a target sight line data determination submodule, configured to determine the acquired sight line data as the user's target sight line data for the calibration mark if the number of recorded displayed calibration marks reaches the preset number; The device further comprises: A data cleaning module, configured to clean the target sight line data to obtain cleaned target sight line data; The target sight line coordinate determination module includes: A target sight point coordinate submodule is configured to determine the target sight point coordinates of the user for the calibration mark in the cockpit camera coordinate system based on the cleaned target sight data; The sight line data includes a plurality of sight line position coordinates of the user captured by the cockpit camera within a preset time period after the calibration mark is displayed; The data cleaning module includes: A first cleaning submodule is configured to cluster the plurality of sight line position coordinates to determine a cluster center; and determine the cluster center as the cleaned target sight line data; A second cleaning submodule is configured to cluster the plurality of sight line position coordinates to determine a category including the largest number of sight line position coordinates; and determine the sight line position coordinates included in the determined category as cleaned target sight line data; A third cleaning submodule is configured to determine a sight focus set of the user based on a capture time sequence corresponding to the plurality of sight position coordinates; and determine the end point position coordinates of the sight focus set as the cleaned target sight data; a fourth cleaning submodule, configured to determine a sight focus set of the user based on a capture time sequence corresponding to the plurality of sight position coordinates; and determine a sight position coordinate whose distance from an end position coordinate of the sight focus set is less than a preset threshold as the cleaned target sight data; The device further comprises: A current sight line data acquisition module, configured to acquire the current sight line data of the user; a sight line data calibration module, configured to determine the product of the sight line calibration matrix and the current sight line data as calibrated sight line data; The device further comprises: A sight calibration prompt information display module, used to control the projection area of ​​the cockpit to display a sight calibration prompt information, wherein the sight calibration prompt information is used to prompt the user to look at the calibration mark displayed in the projection area; a sight calibration prompt voice playback module, configured to control a cockpit audio output device to play a sight calibration prompt voice, wherein the sight calibration prompt voice is used to prompt the user to look at the calibration mark displayed in the projection area; The calibration mark is displayed in a projection area of ​​a head-up display (HUD) in the cockpit.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images