Driver sight display method and system, computer equipment and storage medium

Abstract

The invention provides a driver sight display method and system, computer equipment and a storage medium, and the method comprises the steps: collecting a cab image of a vehicle cab in real time through a vehicle-mounted camera after a driver enters the vehicle cab; the vehicle terminal determines a three-dimensional direction vector of a first sight line direction of a sight line drop point of the driver in a vehicle-mounted camera coordinate system according to the cab image, and converts the three-dimensional direction vector of the first sight line direction to an image coordinate system to obtain a second sight line drop point coordinate; and the management end draws a driver sight drop point diagram according to the second sight drop point coordinates. By adopting the scheme, the sight drop point of the driver is accurately obtained, and the sight state of the driver is monitored.

Description

Technical Field

[0001] The present invention relates to the field of intelligent automobile cockpits, and in particular to a driver's line of sight display method, system, computer equipment and storage medium. Background Art

[0002] With the rapid development of the automobile industry, the functional complexity of vehicles is increasing, and the regulatory requirements for driver and passenger monitoring, safe driving, etc. are becoming increasingly stringent. In the prior art, in order to improve the safety of vehicle driving, it is necessary to monitor the driver's driving state so as to safely control the vehicle according to the driver's driving state.

[0003] However, the study found that because the driver's hand or head movements usually have a large range and amplitude when driving a vehicle, it is easier to monitor the changes in the state of his hands or head when monitoring his driving status. However, for parts such as eyes, nose, tongue, etc. that are smaller in size, amplitude, and range of movement, it is more difficult to monitor these parts, and the accuracy of the monitored status results is also low, especially the changes in the line of sight state caused by eye movements. It is even more difficult to monitor. The driver's line of sight state is indispensable in safety detection such as distraction detection and fatigue detection. Therefore, how to obtain the driver's line of sight and monitor the driver's line of sight state has become a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, system, computer device and storage medium for displaying the driver's line of sight, so as to accurately obtain the driver's line of sight and monitor the driver's line of sight status.

[0005] In a first aspect, an embodiment of the present application provides a driver's line of sight display method, which is applied to a driver's line of sight display system, wherein the system includes a vehicle-mounted camera, a vehicle terminal, and a management terminal, and the method includes:

[0006] After the driver enters the vehicle cab, the vehicle-mounted camera collects a cab image of the vehicle cab in real time;

[0007] The vehicle computer determines the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image, and converts the three-dimensional vector of the first sight direction into the image coordinate system to obtain the coordinates of the second sight point;

[0008] The management end draws a driver's sight point map according to the second sight point coordinates.

[0009] Optionally, determining the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image includes:

[0010] Determine a facial region image of the driver in the cab image based on a face detection model;

[0011] Determine the coordinates of several facial feature points of the driver using a facial feature point extraction model according to the facial region image;

[0012] The images of the left eye and the right eye are cut out from the face image according to the coordinates of each feature point indicating the eye part among the coordinates of a plurality of facial feature points, and the three-dimensional direction vectors of the sight lines of the left eye and the right eye are determined by using a sight line direction estimation model.

[0013] Optionally, the extracting the left eye and right eye images from the face image according to the coordinates of each feature point indicating the eye part among the plurality of facial feature point coordinates, and determining the three-dimensional direction vectors of the sight lines of the left eye and the right eye using a sight line direction estimation model, comprises:

[0014] Determining the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye from the coordinates of a plurality of facial feature points according to the sequence number of each facial feature point;

[0015] The coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye are cut out from the face image to obtain the left eye and the right eye images, and the left eye image and the right eye image are respectively input into the trained sight line estimation model to obtain the three-dimensional vector coordinates of the left eye sight line direction of the left eye and the three-dimensional vector coordinates of the right eye sight line direction of the right eye;

[0016] The three-dimensional direction vector of the first sight direction is obtained by calculating the average value of the three-dimensional vector coordinates of the left eye sight direction and the three-dimensional vector coordinates of the right eye sight direction.

[0017] Optionally, converting the three-dimensional direction vector of the first sight line direction into an image coordinate system to obtain the coordinates of the second sight line landing point includes:

[0018] The three-dimensional direction vector of the first sight line direction is converted into the virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector;

[0019] The coordinates of the second sight line landing point are obtained by projecting the virtual camera sight line direction vector coordinates into the image coordinate system according to the virtual camera intrinsic parameter matrix.

[0020] Optionally, the virtual camera external parameter matrix includes a rotation matrix and a translation vector; the virtual camera internal parameter matrix includes a focal length and an optical center coordinate;

[0021] The step of converting the three-dimensional direction vector of the first sight line direction into a virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector includes:

[0022] The virtual camera sight direction vector coordinate d is determined according to the following expression: v :

[0023] d v =R v ·g c ;

[0024] Among them, R v is the rotation matrix, g c is the three-dimensional direction vector of the first sight direction;

[0025] The step of projecting the coordinates of the virtual camera sight direction vector to the image coordinate system according to the virtual camera intrinsic parameter matrix to obtain the coordinates of the second sight point includes:

[0026] The coordinates P of the second sight point are determined according to the following expression: vi :

[0027]

[0028] Among them, x vi The coordinates of the second sight point P vi The coordinate value in the x-axis direction of the image coordinate system, y vi The coordinates of the second sight point P vi is the coordinate value in the y-axis direction of the image coordinate system, f is the focal length, cu is the coordinate value of the optical center coordinate in the x-axis direction of the virtual camera coordinate system, cv is the coordinate value of the optical center coordinate in the y-axis direction of the virtual camera coordinate system, x v is the virtual camera sight direction vector coordinate, denoted by d v The coordinate value in the x-axis direction of the virtual camera coordinate system, y v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the y-axis direction of the virtual camera coordinate system, z v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the z-axis direction of the virtual camera coordinate system.

[0029] Optionally, the method further comprises:

[0030] The vehicle terminal obtains the first region of interest coordinates of each region of interest in the vehicle cab in the vehicle-mounted camera coordinate system;

[0031] The vehicle computer terminal converts the first region of interest coordinates of each region of interest into the image coordinate system to obtain the second region of interest coordinates of each region of interest;

[0032] The management end draws a vehicle cab region of interest map according to the second region of interest coordinates of each region of interest.

[0033] Optionally, the method further comprises:

[0034] The management end displays the driver's sight point map and / or the vehicle cab area of ​​interest map through a display interface.

[0035] In a second aspect, an embodiment of the present application provides a driver's sight display system, the system comprising a vehicle-mounted camera, a vehicle terminal, and a management terminal;

[0036] The vehicle-mounted camera is used to collect a cab image of the vehicle cab in real time after the driver enters the vehicle cab;

[0037] The vehicle computer end is used to determine the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image, and convert the three-dimensional direction vector of the first sight direction into the image coordinate system to obtain the coordinates of the second sight point;

[0038] The management end is used to draw a driver's sight point map according to the second sight point coordinates.

[0039] Optionally, determining the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image includes:

[0040] Determine a facial region image of the driver in the cab image based on a face detection model;

[0041] Determine the coordinates of several facial feature points of the driver using a facial feature point extraction model according to the facial region image;

[0042] The images of the left eye and the right eye are cut out from the face image according to the coordinates of each feature point indicating the eye part among the coordinates of a plurality of facial feature points, and the three-dimensional direction vectors of the sight lines of the left eye and the right eye are determined by using a sight line direction estimation model.

[0043] Optionally, the extracting the left eye and right eye images from the face image according to the coordinates of each feature point indicating the eye part among the plurality of facial feature point coordinates, and determining the three-dimensional direction vectors of the sight lines of the left eye and the right eye using a sight line direction estimation model, comprises:

[0044] Determining the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye from the coordinates of a plurality of facial feature points according to the sequence number of each facial feature point;

[0045] The coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye are cut out from the face image to obtain the left eye and the right eye images, and the left eye image and the right eye image are respectively input into the trained sight line estimation model to obtain the three-dimensional vector coordinates of the left eye sight line direction of the left eye and the three-dimensional vector coordinates of the right eye sight line direction of the right eye;

[0046] The three-dimensional direction vector of the first sight direction is obtained by calculating the average value of the three-dimensional vector coordinates of the left eye sight direction and the three-dimensional vector coordinates of the right eye sight direction.

[0047] Optionally, converting the three-dimensional direction vector of the first sight line direction into an image coordinate system to obtain the coordinates of the second sight line landing point includes:

[0048] The three-dimensional direction vector of the first sight line direction is converted into the virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector;

[0049] The coordinates of the second sight line landing point are obtained by projecting the virtual camera sight line direction vector coordinates into the image coordinate system according to the virtual camera intrinsic parameter matrix.

[0050] Optionally, the virtual camera external parameter matrix includes a rotation matrix and a translation vector; the virtual camera internal parameter matrix includes a focal length and an optical center coordinate;

[0051] The step of converting the three-dimensional direction vector of the first sight line direction into a virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector includes:

[0052] The virtual camera sight direction vector coordinate d is determined according to the following expression: v :

[0053] d v =R v ·g c ;

[0054] Among them, R v is the rotation matrix, g c is the three-dimensional direction vector of the first sight direction;

[0055] The step of projecting the coordinates of the virtual camera sight direction vector to the image coordinate system according to the virtual camera intrinsic parameter matrix to obtain the coordinates of the second sight point includes:

[0056] The coordinates P of the second sight point are determined according to the following expression: vi :

[0057]

[0058] Among them, x vi The coordinates of the second sight point P vi The coordinate value in the x-axis direction of the image coordinate system, y vi The coordinates of the second sight point P vi is the coordinate value in the y-axis direction of the image coordinate system, f is the focal length, cu is the coordinate value of the optical center coordinate in the x-axis direction of the virtual camera coordinate system, cv is the coordinate value of the optical center coordinate in the y-axis direction of the virtual camera coordinate system, x v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the x-axis direction of the virtual camera coordinate system, y v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the y-axis direction of the virtual camera coordinate system, z v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the z-axis direction of the virtual camera coordinate system.

[0059] Optionally, the vehicle terminal is further used to obtain the coordinates of the first region of interest of each region of interest in the vehicle cab in the vehicle-mounted camera coordinate system;

[0060] The vehicle computer end is further used to convert the first region of interest coordinates of each region of interest into the image coordinate system to obtain the second region of interest coordinates of each region of interest;

[0061] The management end is further used to draw a vehicle cab area of ​​interest map according to the second area of ​​interest coordinates of each area of ​​interest.

[0062] Optionally, the management end is further used to display the driver's line of sight map and / or the vehicle cab area of ​​interest map through a display interface.

[0063] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the driver's line of sight display method described in any optional implementation manner of the first aspect are performed.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the driver's line of sight display method described in any optional implementation manner in the above-mentioned first aspect are executed.

[0065] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0066] This application uses a driver sight display system. After the driver enters the vehicle cab, the vehicle cab image is captured in real time through the on-board camera to obtain an image containing the driver's face as the basis for subsequent driver sight analysis.

[0067] Then, the vehicle computer determines the three-dimensional direction vector of the first line of sight direction of the driver's line of sight in the vehicle-mounted camera coordinate system based on the cab image, and converts the three-dimensional direction vector of the first line of sight direction to the image coordinate system to obtain the coordinates of the second line of sight point. The position of the driver's line of sight can be determined based on the cab image to obtain the driver's line of sight state.

[0068] Finally, the management end draws a driver's sight point map according to the second sight point coordinates to monitor the driver's sight state.

[0069] By adopting the above method, the image of the vehicle cab is collected through the driver's line of sight display system, the driver's line of sight state is determined based on the driver's facial information in the collected image, and finally the driver's line of sight map is drawn based on the line of sight state to monitor the driver's line of sight state.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0072] Figure 1 A flowchart of a method for displaying a driver's sight line provided by the first embodiment of the present invention is shown;

[0073] Figure 2 A flowchart of a method for determining a three-dimensional direction vector of a first sight line direction provided by Embodiment 1 of the present invention is shown;

[0074] Figure 3 A flowchart of a specific method for determining a three-dimensional direction vector of a first sight line direction provided by the first embodiment of the present invention is shown;

[0075] Figure 4 A flow chart of a method for determining the coordinates of a second sight point provided by the first embodiment of the present invention is shown;

[0076] Figure 5 A flow chart of a method for generating a region of interest map of a vehicle cab provided by a first embodiment of the present invention is shown;

[0077] Figure 6 A schematic diagram of the structure of a driver's sight line display system provided by the second embodiment of the present invention is shown;

[0078] Figure 7 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0080] Embodiment 1

[0081] To facilitate understanding of this application, Figure 1 The contents described in the flowchart of a method for displaying a driver's line of sight provided in the first embodiment of the present invention are described in detail in the first embodiment of the present application.

[0082] See also Figure 1 As shown, Figure 1 A flowchart of a method for displaying a driver's line of sight provided by a first embodiment of the present invention is shown, wherein the method is applied to a driver's line of sight display system, the system comprising a vehicle-mounted camera, a vehicle terminal and a management terminal, and the method comprises steps S101 to S103:

[0083] S101: After the driver enters the vehicle cab, the vehicle-mounted camera collects a cab image of the vehicle cab in real time.

[0084] Specifically, the vehicle-mounted camera is set in the vehicle cab, and the specific setting position is such that its field of view can cover the position of the driver's head area when driving the vehicle. In the driver's sight display system, the vehicle-mounted camera and the vehicle-machine end can communicate, and the vehicle-machine end and the management end can communicate. After the driver enters the vehicle cab, the vehicle is started, and the driver's sight display system is started, and the vehicle-mounted camera is used to collect the cab image of the vehicle cab in real time. Since the vehicle-mounted camera is set at a position where its field of view can cover the driver's head area when driving the vehicle, the collected cab image contains the driver's head area.

[0085] S102: The vehicle computer determines the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system based on the cab image, and converts the three-dimensional direction vector of the first sight direction to the image coordinate system to obtain the coordinates of the second sight point.

[0086] Specifically, the vehicle computer end determines a head image indicating the driver's head area from the cab image; the vehicle computer end determines an eye image (including a left eye image and a right eye image) indicating the driver's eye area from the head image indicating the driver's head area; the vehicle computer end determines the driver's left eye sight and right eye sight based on the image information of the left eye image and the right eye image indicating the driver, respectively; the vehicle computer end determines the driver's comprehensive sight point, that is, the three-dimensional direction vector of the first sight direction, based on the driver's left eye sight and right eye sight.

[0087] Since the cab image is collected by the vehicle-mounted camera, the image information is generated in the vehicle-mounted camera coordinate system, so the coordinate information of the sight point is also in the vehicle-mounted camera coordinate system (three-dimensional space coordinate system), that is, the three-dimensional direction vector of the first sight direction. In order to draw the sight point in the plane diagram, the three-dimensional direction vector of the first sight direction needs to be converted to the image coordinate system (two-dimensional plane coordinate system) to obtain the coordinates of the second sight point.

[0088] S103: The management terminal draws a driver's sight point map according to the second sight point coordinates.

[0089] Specifically, the management end generates a plane rectangular coordinate system according to the size of its display interface, and draws the sight point indicated by the second sight point coordinate in the plane rectangular coordinate system to obtain the driver's sight point map. Since the on-board camera collects images in real time and transmits them to the vehicle computer end for sight point determination, the vehicle computer end sends its sight point coordinates to the management end in real time, and the management end draws the driver's sight point map according to its real-time sight point. The final display form is a dynamic sight point map, so that the management end can monitor the driver's sight point according to the driver's sight point map.

[0090] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flowchart of a method for determining a three-dimensional direction vector of a first sight line direction provided by a first embodiment of the present invention is shown, wherein the method for determining the three-dimensional direction vector of the first sight line direction of the driver's sight line in the vehicle-mounted camera coordinate system according to the cab image comprises steps S201 to S203:

[0091] S201: Determine a facial area image of the driver in the cab image based on a face detection model.

[0092] Specifically, the face detection model is a trained CNN convolutional neural network model that can recognize faces or heads. The face detection model detects the rectangular frame position of the driver's face in the cab image, and cuts out the head or face area according to the rectangular frame position of the head, that is, the facial area image.

[0093] S202: Determine the coordinates of several facial feature points of the driver using a facial feature point extraction model based on the facial region image.

[0094] Specifically, the facial region image is input into the trained facial feature point extraction model to infer the two-dimensional coordinate information of 68 facial feature points (standard facial feature points) of the driver's face on the facial region image, that is, the coordinates of several facial feature points.

[0095] S203: extracting the left eye and right eye images from the face image according to the coordinates of each feature point indicating the eye part among the plurality of facial feature point coordinates, and determining the three-dimensional direction vectors of the sight lines of the left eye and the right eye using a sight line direction estimation model.

[0096] Specifically, the direction vector of the driver's left eye sight is determined according to the coordinates of each feature point indicating the left eye part, and the direction vector of the driver's right eye sight is determined according to the coordinates of each feature point indicating the right eye part; the three-dimensional direction vector of the driver's first sight direction is determined according to the direction vector of the driver's left eye sight and the direction vector of the driver's right eye sight.

[0097] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flowchart of a specific method for determining a three-dimensional direction vector of a first sight line direction provided by the first embodiment of the present invention is shown, wherein the method includes extracting the images of the left eye and the right eye from the face image according to the coordinates of each feature point indicating the eye part in the coordinates of a plurality of facial feature points, and determining the three-dimensional direction vectors of the sight line directions of the left eye and the right eye using a sight line direction estimation model, including steps S301 to S303:

[0098] S301: Determine, from among a plurality of facial feature point coordinates, the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye according to the sequence number of each facial feature point.

[0099] Specifically, each of the 68 standard feature points on the driver's face is marked with a serial number, and the feature points numbered 42 to 47 (indicating the left eye) and the feature points numbered 36 to 41 (indicating the right eye) are extracted to obtain the coordinates of the feature points indicating the left eye and the coordinates of the feature points indicating the right eye, respectively.

[0100] S302: Extract the left eye and right eye images from the face image using the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye, and input the left eye image and the right eye image into the trained line of sight estimation model respectively to obtain the three-dimensional vector coordinates of the left eye line of sight direction of the left eye and the three-dimensional vector coordinates of the right eye line of sight direction of the right eye.

[0101] Specifically, a left eye region image indicating the left eye and a right eye region image indicating the right eye are cropped from the facial region image and input into the sight line estimation model to obtain the left eye sight line coordinate g lc and the right eye sight coordinate g rc .

[0102] S303: Calculate the average value of the three-dimensional vector coordinates of the left eye sight direction and the three-dimensional vector coordinates of the right eye sight direction to obtain the three-dimensional direction vector of the first sight direction.

[0103] Specifically, the average value of the left eye sight coordinates and the right eye sight coordinates is calculated to obtain the three-dimensional direction vector of the first sight direction. Here, the average value of the left eye sight coordinates and the right eye sight coordinates within a period of time can also be calculated to improve the stability of the calculation.

[0104] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flow chart of a method for determining the coordinates of a second sight point provided by the first embodiment of the present invention is shown, wherein the method of converting the three-dimensional direction vector of the first sight direction into the image coordinate system to obtain the coordinates of the second sight point includes steps S401 to S402:

[0105] S401: Convert the three-dimensional direction vector of the first sight direction into a virtual camera coordinate system according to a virtual camera extrinsic parameter matrix to obtain coordinates of the virtual camera sight direction vector.

[0106] Specifically, since the ultimate goal is to enable the management end to view the driver's sight point in real time from the perspective of the head posture, a virtual camera is introduced. The virtual camera is a camera that uses the current driver's eyes as the observation perspective, and the virtual camera coordinate system is constructed with the midpoint of the driver's eyes as the origin. First, the three-dimensional direction vector of the first sight direction in the vehicle-mounted camera coordinate system is converted to the virtual camera coordinate system to obtain the virtual camera sight direction vector coordinates, completing the coordinate conversion under different views in the same three-dimensional space.

[0107] S402: Projecting the coordinates of the sight line direction vector of the virtual camera to the image coordinate system according to the virtual camera intrinsic parameter matrix to obtain the coordinates of the second sight line landing point.

[0108] Specifically, the virtual camera sight line direction vector coordinates in the virtual camera coordinate system (three-dimensional space coordinate system) are then projected to the image coordinate system (two-dimensional plane coordinate system) to obtain the second sight line landing point coordinates, which is convenient for subsequent drawing of the two-dimensional plane image.

[0109] In an optional implementation, the virtual camera external parameter matrix includes a rotation matrix and a translation vector; the virtual camera internal parameter matrix includes a focal length and an optical center coordinate.

[0110] Specifically, the extrinsic matrix A of the virtual camera is v for Among them, R v is the rotation matrix, T v is the translation vector, R v and the driver's head posture matrix A c The rotation matrix in is the same as R v =R c ; Since the observation angles of the virtual camera and the human eye are consistent, the conversion value of the translation vector is the starting point of the driver's line of sight S c , that is, T v =S c Therefore, v Reassignment can be obtained

[0111] Head pose matrix A c It is obtained by estimating the facial region image and the facial feature points of the driver. Specifically, the facial region image and the facial feature points of the driver are input into the trained head posture estimation model to output the head posture matrix of the driver's head in the camera coordinate system. The head posture matrix is ​​composed of the head rotation matrix R c and the translation vector T of the head c Therefore, the head posture matrix A c Expressed as

[0112] The intrinsic parameter matrix I of the virtual camera v Including the focal length f and the optical center (or principal point) coordinates (cu, cv), calculated from the width w, height h, and vertical field of view angle θ of the management terminal display interface, I v Expressed as in:

[0113]

[0114]

[0115] In practical applications, the width w, height h, and vertical field of view angle θ can be selected from a set of appropriate values ​​according to the presentation effect, such as w is 800 pixels, h is 600 pixels, and θ is 90 degrees.

[0116] The step of converting the three-dimensional direction vector of the first sight line direction into a virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector includes:

[0117] The virtual camera sight direction vector coordinate d is determined according to the following expression: v :

[0118] d v =R v ·g c ;

[0119] Among them, R v is the rotation matrix, g c is the three-dimensional direction vector of the first sight direction.

[0120] The step of projecting the coordinates of the virtual camera sight direction vector to the image coordinate system according to the virtual camera intrinsic parameter matrix to obtain the coordinates of the second sight point includes:

[0121] The coordinates P of the second sight point are determined according to the following expression: vi :

[0122]

[0123] Among them, x vi The coordinates of the second sight point P vi The coordinate value in the x-axis direction of the image coordinate system, y vi The coordinates of the second sight point P vi is the coordinate value in the y-axis direction of the image coordinate system, f is the focal length, cu is the coordinate value of the optical center coordinate in the x-axis direction of the virtual camera coordinate system, cv is the coordinate value of the optical center coordinate in the y-axis direction of the virtual camera coordinate system, x v is the vector coordinate of the virtual camera’s sight direction, dv The coordinate value in the x-axis direction of the virtual camera coordinate system, y v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the y-axis direction of the virtual camera coordinate system, z v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the z-axis direction of the virtual camera coordinate system.

[0124] In an alternative embodiment, see Figure 5 As shown, Figure 5 A flowchart of a method for generating a vehicle cab region of interest map provided by the first embodiment of the present invention is shown, wherein the method further includes steps S501 to S503:

[0125] S501: The vehicle computer obtains the first region of interest coordinates of each region of interest in the vehicle cab in the vehicle-mounted camera coordinate system.

[0126] Specifically, the vehicle computer extracts the first area of ​​interest coordinates of each AOI (Area of ​​Interest, GazeAOI can be described as the gaze area) area in the vehicle camera coordinate system from the cockpit digital model, or uses real vehicle modeling to obtain the first area of ​​interest coordinates of each area of ​​interest in the vehicle camera coordinate system.

[0127] The areas of interest include various functional areas in the vehicle cab, such as doors, rearview mirrors, steering wheels, and instrument panels.

[0128] Before obtaining the coordinates of the first region of interest of each region of interest in the vehicle cab in the vehicle camera coordinate system, since the vehicle camera can be set at different positions in the vehicle (such as the A-pillar, instrument screen, steering wheel column position, etc.), when the camera is installed on a movable part in the vehicle, the coordinate system must be converted and unified. For example, if the camera is set on the steering wheel column, when the driver stretches forward and backward and rotates the steering wheel up and down to adjust the steering wheel, the position of the vehicle camera will change with the adjustment of the steering wheel, that is, the position external parameter of the vehicle camera coordinate system in the cockpit changes, and the coordinates of each region of interest are inconsistent with the vehicle camera, so it is necessary to convert the AOI coordinates to the vehicle camera coordinate system at the current position to obtain the coordinates of each first region of interest.

[0129] S502: The vehicle computer terminal converts the first region-of-interest coordinates of each region of interest into the image coordinate system to obtain the second region-of-interest coordinates of each region of interest.

[0130] Specifically, referring to the conversion process of the driver's line of sight coordinates in steps S401 to S402, the first region of interest coordinates of each region of interest are first converted to the virtual camera coordinate system to obtain the candidate region of interest coordinates, and then the candidate region of interest coordinates are converted to the image coordinate system to obtain the second region of interest coordinates of each region of interest.

[0131] The process of converting the coordinates of the first region of interest of each region of interest to the virtual camera coordinate system to obtain the coordinates of the candidate region of interest is as follows: multiply the coordinates of the first region of interest of each region of interest by the external parameter matrix A of the virtual camera on the left v Convert to the virtual camera coordinate system to obtain the candidate region of interest coordinates, that is, determine the candidate region of interest coordinates P by the following expression vaoi :

[0132] P vaoi =A v ·P caoi ;

[0133] Among them, A v is the extrinsic matrix of the virtual camera, P caoi is the coordinate of the first region of interest.

[0134] The process of converting the coordinates of each candidate region of interest into the image coordinate system to obtain the coordinates of the second region of interest of each region of interest is as follows: the coordinates P of each second region of interest are determined according to the following expression: iaoi :

[0135]

[0136] Among them, x iaoi is the coordinates of the second region of interest P iaoi The coordinate value in the x-axis direction of the image coordinate system, y iaoi is the coordinates of the second region of interest P iaoi is the coordinate value in the y-axis direction of the image coordinate system, f is the focal length, cu is the coordinate value of the optical center coordinate in the x-axis direction of the virtual camera coordinate system, cv is the coordinate value of the optical center coordinate in the y-axis direction of the virtual camera coordinate system, x vaoi is the coordinates of the candidate region of interest P vaoi The coordinate value in the x-axis direction of the virtual camera coordinate system, y vaoi is the coordinates of the candidate region of interest P vaoi The coordinate value in the y-axis direction of the virtual camera coordinate system, z vaoi is the coordinates of the candidate region of interest P vaoi The coordinate value in the z-axis direction of the virtual camera coordinate system.

[0137] S503: The management terminal draws a vehicle cab region of interest map according to the second region of interest coordinates of each region of interest.

[0138] Specifically, the management end generates a plane rectangular coordinate system, and for each region of interest, the points indicated by the coordinates of the second region of interest in the plane rectangular coordinate system are drawn in the coordinate system, and then all the points in the coordinate system are connected with straight lines in a clockwise or counterclockwise direction to obtain a map of the region of interest in the vehicle cab.

[0139] In an optional embodiment, the method further comprises:

[0140] The management end displays the driver's sight point map and / or the vehicle cab area of ​​interest map through a display interface.

[0141] Specifically, drawing rules are pre-configured in the management terminal, and the drawing rules include but are not limited to color, line thickness, fill type, etc. The management terminal displays the driver's sight point map and / or the vehicle cab area of ​​interest map based on these drawing rules. Further, for different areas of interest, lines indicating different areas of interest are filled and drawn with lines of different colors. When the driver's sight point map and the vehicle cab area of ​​interest map are displayed at the same time, the driver's sight point is filled with a color different from the fill color of the line of the area of ​​interest to distinguish different areas and different types of points.

[0142] In addition, the driver's left eye sight point and right eye sight point can also be drawn. For example, a blue dot is used to represent the left eye sight point, a pink dot is used to represent the right eye sight point, a red dot is used to represent the coordinates of the second sight point, and other color line boxes represent areas of interest.

[0143] During the operation of the driver's sight display system, if the vehicle terminal has a display interface, the driver's sight point map and / or the vehicle cab area of ​​interest map can be displayed on the vehicle terminal. Considering that when the vehicle terminal has no display interface, after the vehicle terminal calculates the coordinates of the second sight point and the coordinates of the second area of ​​interest of each area of ​​interest, each coordinate will be written into the drawing command and sent to the management terminal through the network. After receiving the drawing command, the management terminal draws the sight point and the area of ​​interest. During the drawing process, the content that can be drawn includes but is not limited to lines, circles and texts. Drawing lines requires knowing the starting and ending coordinates of the line, the color and type of the line, etc. Drawing points requires knowing the center coordinates of the point, color, radius, fill and other information, and texts require knowing the text content, position, color, etc., so that the drawing command group can be converted into a json object and sent to the management terminal. The management terminal parses the json object of the drawing command, draws and displays the image, so that the management terminal can observe the sight point and the area of ​​interest in front of the driver in real time.

[0144] In the above technical solution, the vehicle computer mainly calculates a virtual human eye perspective camera based on the head posture for display. As an optimized solution, the head posture and the line of sight direction are combined to calculate the virtual human eye line of sight camera for display.

[0145] Embodiment 2

[0146] See also Figure 6 As shown, Figure 6 The structure diagram of a driver sight display system provided by the second embodiment of the present invention is shown, wherein the system includes a vehicle-mounted camera 601, a vehicle terminal 602 and a management terminal 603;

[0147] The vehicle-mounted camera is used to collect a cab image of the vehicle cab in real time after the driver enters the vehicle cab;

[0148] The vehicle computer end is used to determine the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image, and convert the three-dimensional direction vector of the first sight direction into the image coordinate system to obtain the coordinates of the second sight point;

[0149] The management end is used to draw a driver's sight point map according to the second sight point coordinates.

[0150] In an optional implementation, determining the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image includes:

[0151] Determine a facial region image of the driver in the cab image based on a face detection model;

[0152] Determine the coordinates of several facial feature points of the driver using a facial feature point extraction model according to the facial region image;

[0153] The three-dimensional direction vector of the first sight line direction is determined according to the coordinates of each feature point indicating the eye part among the coordinates of the plurality of facial feature points.

[0154] In an optional implementation, determining the three-dimensional direction vector of the first sight line direction according to the coordinates of each feature point indicating the eye part among the coordinates of a plurality of facial feature points includes:

[0155] Determining the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye from the coordinates of a plurality of facial feature points according to the sequence number of each facial feature point;

[0156] Inputting the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye into the trained sight line estimation model respectively to obtain the left eye sight line coordinates of the left eye and the right eye sight line coordinates of the right eye;

[0157] The three-dimensional direction vector of the first sight direction is obtained by calculating the average value of the left eye sight coordinates and the right eye sight coordinates.

[0158] In an optional implementation, converting the three-dimensional direction vector of the first sight direction into an image coordinate system to obtain the coordinates of the second sight point includes:

[0159] The three-dimensional direction vector of the first sight line direction is converted into the virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector;

[0160] The coordinates of the second sight line landing point are obtained by projecting the virtual camera sight line direction vector coordinates into the image coordinate system according to the virtual camera intrinsic parameter matrix.

[0161] In an optional embodiment, the virtual camera external parameter matrix includes a rotation matrix and a translation vector; the virtual camera internal parameter matrix includes a focal length and an optical center coordinate;

[0162] The step of converting the three-dimensional direction vector of the first sight line direction into a virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector includes:

[0163] The virtual camera sight direction vector coordinate d is determined according to the following expression: v :

[0164] d v =R v ·g c ;

[0165] Among them, R v is the rotation matrix, g c is the three-dimensional direction vector of the first sight direction;

[0166] The step of projecting the coordinates of the virtual camera sight direction vector to the image coordinate system according to the virtual camera intrinsic parameter matrix to obtain the coordinates of the second sight point includes:

[0167] The coordinates P of the second sight point are determined according to the following expression: vi :

[0168]

[0169] Among them, x vi The coordinates of the second sight point P vi The coordinate value in the x-axis direction of the image coordinate system, y vi The coordinates of the second sight point P vi is the coordinate value in the y-axis direction of the image coordinate system, f is the focal length, cu is the coordinate value of the optical center coordinate in the x-axis direction of the virtual camera coordinate system, cv is the coordinate value of the optical center coordinate in the y-axis direction of the virtual camera coordinate system, x v is the vector coordinate d of the virtual camera's sight direction v The coordinate value in the x-axis direction of the virtual camera coordinate system, y v is the vector coordinate d of the virtual camera's sight direction v The coordinate value in the y-axis direction of the virtual camera coordinate system, z v is the vector coordinate d of the virtual camera's sight direction v The coordinate value in the z-axis direction of the virtual camera coordinate system.

[0170] In an optional implementation, the vehicle terminal is further used to obtain the coordinates of the first region of interest of each region of interest in the vehicle cab in the vehicle-mounted camera coordinate system;

[0171] The vehicle computer end is further used to convert the first region of interest coordinates of each region of interest into the image coordinate system to obtain the second region of interest coordinates of each region of interest;

[0172] The management end is further used to draw a vehicle cab area of ​​interest map according to the second area of ​​interest coordinates of each area of ​​interest.

[0173] In an optional implementation, the management terminal is also used to display the driver's sight point map and / or the vehicle cab area of ​​interest map through a display interface.

[0174] Embodiment 3

[0175] Based on the same application concept, see Figure 7 As shown, Figure 7 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Figure 7 As shown, a computer device 700 provided in Embodiment 3 of the present application includes:

[0176] A processor 701, a memory 702 and a bus 703, wherein the memory 702 stores machine-readable instructions executable by the processor 701. When the computer device 700 is running, the processor 701 communicates with the memory 702 via the bus 703. When the processor 701 is running, the machine-readable instructions execute the steps of the driver's line of sight display method shown in the above-mentioned embodiment 1.

[0177] Embodiment 4

[0178] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the driver's line of sight display method described in any of the above embodiments are executed.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0180] The computer program product for displaying the driver's line of sight provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0181] The driver sight display system provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0182] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0183] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0185] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0186] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0187] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for displaying a driver's sight line, characterized in that: Applied to a driver's sight display system, the system includes a vehicle-mounted camera, a vehicle terminal, and a management terminal, and the method includes: After the driver enters the vehicle cab, the vehicle-mounted camera collects a cab image of the vehicle cab in real time; The vehicle computer determines the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image, and converts the three-dimensional direction vector of the first sight direction into the image coordinate system to obtain the coordinates of the second sight point; The management end draws a driver's sight point map according to the second sight point coordinates.

2. The method according to claim 1, characterized in that The method of determining, based on the cab image, a three-dimensional direction vector of a first sight direction of the driver's sight point in a vehicle-mounted camera coordinate system includes: Determine a facial region image of the driver in the cab image based on a face detection model; Determine the coordinates of several facial feature points of the driver using a facial feature point extraction model according to the facial region image; The images of the left eye and the right eye are cut out from the face image according to the coordinates of each feature point indicating the eye part among the coordinates of a plurality of facial feature points, and the three-dimensional direction vectors of the sight lines of the left eye and the right eye are determined by using a sight line direction estimation model.

3. The method according to claim 2, characterized in that The method includes: extracting the left eye and the right eye images from the face image according to the coordinates of each feature point indicating the eye part in the coordinates of the plurality of facial feature points, and determining the three-dimensional direction vectors of the sight lines of the left eye and the right eye by using the sight line direction estimation model, including: Determining the coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye from the coordinates of a plurality of facial feature points according to the sequence number of each facial feature point; The coordinates of each feature point indicating the left eye and the coordinates of each feature point indicating the right eye are cut out from the face image to obtain the left eye and the right eye images, and the left eye image and the right eye image are respectively input into the trained sight line estimation model to obtain the three-dimensional vector coordinates of the left eye sight line direction of the left eye and the three-dimensional vector coordinates of the right eye sight line direction of the right eye; The three-dimensional direction vector of the first sight direction is obtained by calculating the average value of the three-dimensional vector coordinates of the left eye sight direction and the three-dimensional vector coordinates of the right eye sight direction.

4. The method according to claim 1, characterized in that: The step of converting the three-dimensional direction vector of the first sight line direction into the image coordinate system to obtain the coordinates of the second sight line landing point includes: The three-dimensional direction vector of the first sight line direction is converted into the virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector; The coordinates of the second sight line landing point are obtained by projecting the virtual camera sight line direction vector coordinates into the image coordinate system according to the virtual camera intrinsic parameter matrix.

5. The method according to claim 4, characterized in that The virtual camera external parameter matrix includes a rotation matrix and a translation vector; the virtual camera internal parameter matrix includes a focal length and an optical center coordinate; The step of converting the three-dimensional direction vector of the first sight line direction into a virtual camera coordinate system according to the virtual camera extrinsic parameter matrix to obtain the coordinates of the virtual camera sight line direction vector includes: The virtual camera sight direction vector coordinate d is determined according to the following expression: v : d v =R v ·g c ; Among them, R v is the rotation matrix, g c is the three-dimensional direction vector of the first sight direction; The step of projecting the coordinates of the virtual camera sight direction vector to the image coordinate system according to the virtual camera intrinsic parameter matrix to obtain the coordinates of the second sight point includes: The coordinates P of the second sight point are determined according to the following expression: vi : Among them, x vi The coordinates of the second sight point P vi The coordinate value in the x-axis direction of the image coordinate system, y vi The coordinates of the second sight point P vi is the coordinate value in the y-axis direction of the image coordinate system, f is the focal length, cu is the coordinate value of the optical center coordinate in the x-axis direction of the virtual camera coordinate system, cv is the coordinate value of the optical center coordinate in the y-axis direction of the virtual camera coordinate system, x v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the x-axis direction of the virtual camera coordinate system, y v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the y-axis direction of the virtual camera coordinate system, z v is the vector coordinate of the virtual camera’s sight direction, d v The coordinate value in the z-axis direction of the virtual camera coordinate system.

6. The method according to claim 1, characterized in that The method further comprises: The vehicle terminal obtains the first region of interest coordinates of each AOI region of interest of the vehicle cab in the vehicle-mounted camera coordinate system; The vehicle computer terminal converts the first region of interest coordinates of each region of interest into the image coordinate system to obtain the second region of interest coordinates of each region of interest; The management end draws a vehicle cab region of interest map according to the second region of interest coordinates of each region of interest.

7. The method according to claim 6, characterized in that The method further comprises: The management end displays the driver's sight point map and / or the vehicle cab area of ​​interest map through a display interface.

8. A driver's sight display system, characterized in that: The system includes a vehicle-mounted camera, a vehicle terminal, and a management terminal; The vehicle-mounted camera is used to collect a cab image of the vehicle cab in real time after the driver enters the vehicle cab; The vehicle computer end is used to determine the three-dimensional direction vector of the first sight direction of the driver's sight point in the vehicle-mounted camera coordinate system according to the cab image, and convert the three-dimensional direction vector of the first sight direction into the image coordinate system to obtain the coordinates of the second sight point; The management end is used to draw a driver's sight point map according to the second sight point coordinates.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the driver's line of sight display method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the driver's line of sight display method as described in any one of claims 1 to 7.

Images