Method and device for determining line-of-sight vector of person in vehicle

Abstract

The present disclosure relates to a method and an apparatus for determining a line-of-sight vector of a person in a vehicle, where the vehicle is configured with a plurality of cameras, and after a posture of the person is determined, the line-of-sight vector of the person is determined. At least two first cameras in the plurality of cameras can shoot pupil positions of the person, and at least one second camera in the plurality of cameras can shoot a target object watched by the person; the method for determining the sight line vector of the person in the vehicle comprises the following steps: determining a first three-dimensional coordinate of at least one pupil of the person based on image data acquired by at least two first cameras; determining a second three-dimensional coordinate of the target object watched by the person based on the image data acquired by the at least one second camera; and determining a sight line vector of the driver based on the first three-dimensional coordinate and the second three-dimensional coordinate.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of vehicle control. Specifically, the present disclosure relates to a method for determining a line-of-sight vector of a person in a vehicle, a device for determining a line-of-sight vector of a person in a vehicle, and a computer-readable storage medium for implementing the above method for determining a line-of-sight vector of a person in a vehicle. Background Art

[0002] In order to develop a monitoring system inside a vehicle, a reliable reference system is needed at this time to collect the ground truth of the driver's head pose and the coordinates of the object being looked at by the driver's eyes.

[0003] In research and development projects, a reference system inside the vehicle is usually established by means of a set of tracking devices (such as those worn on the driver's head). The entire tracking device is relatively easy to deploy and can generally track the driver's head pose. However, since the number of LED lights installed on the windshield is limited and the driver can only look at these LED lights, the number and position of the targets looked at by the driver are restricted.

[0004] Because the driver can only look at one LED light on the windshield, the range of the line-of-sight vector will also be greatly restricted. For example, when the driver looks to the left, since there are usually no LED lights on the left side of the windshield, the line-of-sight vector at this time cannot be measured. Moreover, if the driver does not look at the windshield but at a target in other positions, the corresponding line-of-sight vector cannot be measured either.

[0005] In addition, the fixation origin measured by this method is not very accurate because this method only approximately obtains the fixation origin by taking the center of the two eye corners.

[0006] The above two points will both affect the accuracy of the determined line-of-sight vector. Summary of the Invention

[0007] In order to solve the technical problems in the prior art, that is, how to accurately measure the driver's line-of-sight vector, the inventors of the present disclosure innovatively thought of using the image recognition technology of a camera to replace the device fixedly worn on the driver's head, and thus can accurately determine the driver's line-of-sight vector.

[0008] Specifically, a first aspect of the present disclosure provides a method for determining a line-of-sight vector of a person in a vehicle, where the vehicle is equipped with a plurality of cameras, and after the pose of the person is determined, at least two first cameras among the plurality of cameras can capture the pupil positions of the person and at least one second camera among the plurality of cameras can capture the target object looked at by the person, and the method includes:

[0009] Determine the first three-dimensional coordinates of at least one pupil of the person based on the image data collected by the at least two first cameras;

[0010] Determine the second three-dimensional coordinates of the target object gazed at by the person based on the image data collected by the at least one second camera; and

[0011] Determine the line-of-sight vector of the driver based on the first three-dimensional coordinates and the second three-dimensional coordinates.

[0012] Among the methods for determining the line-of-sight vector of a person in a vehicle according to the present disclosure, first, the first three-dimensional coordinates of at least one pupil of the person, i.e., the starting point of the line-of-sight vector, are determined by means of the image data collected by the at least two first cameras; then, the second three-dimensional coordinates of the target object gazed at by the person, i.e., the ending point of the line-of-sight vector, are determined by means of the image data collected by the at least one second camera; and finally, the line-of-sight vector of the driver is accurately determined by means of the first three-dimensional coordinates and the second three-dimensional coordinates. In this way, the starting and ending positions of the line-of-sight vector determined are accurately determined, thereby ensuring the accurate determination of the line-of-sight vector. Moreover, the ending point of the line-of-sight recognized by the image recognition technology does not have to rely on the position of a preset LED lamp, but the three-dimensional coordinates of the target object gazed at by the person are determined by means of the image recognition technology of the camera, thereby being able to overcome the limitations in the prior art and being able to determine the line-of-sight vector of the person with fewer restrictions.

[0013] Preferably, in an implementation form according to the present disclosure, the method further includes: calibrating the multiple cameras in the same coordinate system. More preferably, in an implementation form according to the present disclosure, calibrating the multiple cameras in the same coordinate system includes: calibrating the multiple cameras in the same coordinate system by means of a black-and-white checkerboard.

[0014] Optionally, in an implementation form according to the present disclosure, the person includes the driver or a passenger of the vehicle.

[0015] Preferably, in an implementation form according to the present disclosure, the target object is configured as a measuring tool including multiple calibration points, and wherein, the second three-dimensional coordinates are determined as the three-dimensional coordinates at the end of the measuring tool. More preferably, in an implementation form according to the present disclosure, the method further includes: fixing the measuring tool to a preset position by means of a fixing tool.

[0016] In one implementation form according to the present disclosure, determining the second three-dimensional coordinates of the target object gazed at by the person based on the image data collected by the at least one second camera includes: determining the plurality of third three-dimensional coordinates of the plurality of calibration points based on the image data collected by the at least one second camera; and determining the second three-dimensional coordinates based on the physical structure data of the measuring tool and the third three-dimensional coordinates.

[0017] Preferably, in one implementation form according to the present disclosure, the first camera and the second camera are multiplexed with each other.

[0018] In addition, a second aspect of the present disclosure provides an apparatus for determining the line-of-sight vector of a person in a vehicle. The vehicle is configured with a plurality of cameras. After the posture of the person is determined, at least two first cameras among the plurality of cameras can capture the pupil positions of the person, and at least one second camera among the plurality of cameras can capture the target object gazed at by the person. The apparatus includes a memory and a processor. A computer-readable instruction is stored on the memory. When the computer-readable instruction is executed by the processor, the processor is caused to implement the method for determining the line-of-sight vector of a person in a vehicle proposed in the first aspect of the present disclosure.

[0019] Furthermore, a third aspect of the present disclosure provides a computer-readable storage medium. A computer-readable instruction is stored on the computer-readable storage medium. When the computer-readable instruction is executed by a processor, the processor is caused to implement the method for determining the line-of-sight vector of a person in a vehicle proposed in the first aspect of the present disclosure.

[0020] In summary, in the method for determining the line-of-sight vector of a person in a vehicle according to the present disclosure, first, the first three-dimensional coordinates of at least one pupil of the person, that is, the starting point of the line-of-sight vector, are determined by means of the image data collected by the at least two first cameras; then, the second three-dimensional coordinates of the target object gazed at by the person, that is, the end point of the line-of-sight vector, are determined by means of the image data collected by the at least one second camera; finally, the line-of-sight vector of the driver is accurately determined by means of the first three-dimensional coordinates and the second three-dimensional coordinates. In this way, the starting point and the end point positions of the line-of-sight vector determined are accurately determined, so that the accurate determination of the line-of-sight vector can be ensured. Furthermore, the line-of-sight end point recognized by the image recognition technology does not have to rely on the position of a preset LED lamp, but the three-dimensional coordinates of the target object gazed at by the person are determined by means of the image recognition technology of the camera, so that the limitations in the prior art can be eliminated, and the determination of the line-of-sight vector of the person can be carried out with fewer restrictions. Description of the Drawings

[0021] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non - limiting embodiments read in conjunction with the accompanying drawings.

[0022] Figure 1 FIG. 6 shows a flowchart of a method 100 for determining a line - of - sight vector of a person inside a vehicle according to an embodiment of the present disclosure;

[0023] Figure 2 FIG. 7 shows a schematic diagram of an apparatus 200 for determining a line - of - sight vector of a person inside a vehicle according to an embodiment of the present disclosure; and

[0024] Figure 3 FIG. 8 shows a schematic diagram of an apparatus 300 for determining a line - of - sight vector of a person inside a vehicle according to another embodiment of the present disclosure.

[0025] In the figures, throughout the different views, the same or similar reference numerals denote the same or similar devices (modules) or steps. Detailed Embodiments

[0026] The following describes in detail various exemplary embodiments of the present disclosure with reference to the accompanying drawings. Although the following described exemplary methods and apparatuses include software and / or firmware executed on hardware among other components, it should be noted that these examples are merely illustrative and should not be considered restrictive. For example, it is contemplated that any or all of the hardware, software, and firmware components can be implemented exclusively in hardware, exclusively in software, or in any combination of hardware and software. Thus, although the following has described exemplary methods and apparatuses, those skilled in the art should readily understand that the provided examples are not used to limit the manner in which these methods and apparatuses are implemented.

[0027] In addition, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of methods and systems according to various embodiments of the present disclosure. It should be noted that the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented using a dedicated hardware - based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0028] As used in this disclosure, the terms "comprising," "including," and similar terms are open-ended terms, i.e., "including / including but not limited to," indicating that other content may also be included. The term "based on" means "at least partially based on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment," and so on.

[0029] As described above, the methods adopted in the prior art are limited by the number and position of the LED lights that are the end points of the line of sight, the dedicated device that needs to be fixed on the head, and only estimating the starting position that is the starting point of the line of sight. Therefore, the line of sight vectors measured by it not only require dedicated devices, but also the measured range is severely limited. Moreover, the measured line of sight vectors are not accurate. To solve the technical problems in the prior art, that is, how to accurately measure the driver's line of sight vector, the inventors of this disclosure innovatively thought of using the image recognition technology of the camera to replace the device fixedly worn on the driver's head, and thus can accurately determine the driver's line of sight vector.

[0030] In the technical solutions proposed in this disclosure, a novel gaze reference system needs to be used to collect line of sight vectors in a larger range for subsequent training of algorithms such as autonomous driving and evaluation of key performance indicators. The method for determining the line of sight vector of a person in the vehicle on which the algorithm training and key performance indicator evaluation are based will enable the test subject to easily gaze at the target point at any position in the three-dimensional space, while the prior art can only collect the gaze target points on the front windshield, and this defect will be solved. In addition, the prior art can only approximately determine the gaze origin by the center point of the eye corner. According to the method for determining the line of sight vector of a person in the vehicle of this disclosure, the three-dimensional coordinates of the through hole that is the gaze origin must also be measured more accurately, that is, the gaze origin must be measured more accurately to improve the accuracy of the overall gaze direction, that is, the line of sight vector. Moreover, during the entire acquisition process, the SmartTrack device worn by the test subject must fit tightly with the test subject's head after calibration, which will also bring inconvenience to the measurement. And if there is relative movement between the head and the SmartTrack device, it will cause the three-dimensional positions of the facial landmarks (including the eye corners) to not be tracked, and the subsequent video sequence will also be useless for the required measurement.

[0031] Generally speaking, the technical solution of the present disclosure proposes a method for determining the line-of-sight vector of a person in a vehicle. The vehicle is equipped with multiple cameras. After the posture of the person is determined, at least two first cameras among the multiple cameras can capture the pupil positions of the person, and at least one second camera among the multiple cameras can capture the target object being gazed at by the person. The method includes: determining the first three-dimensional coordinates of at least one pupil of the person based on the image data collected by the at least two first cameras; determining the second three-dimensional coordinates of the target object being gazed at by the person based on the image data collected by the at least one second camera; and determining the line-of-sight vector of the driver based on the first three-dimensional coordinates and the second three-dimensional coordinates. Among the methods for determining the line-of-sight vector of a person in a vehicle according to the present disclosure, first, the first three-dimensional coordinates of at least one pupil of the person, that is, the starting point of the line-of-sight vector, are determined by means of the image data collected by the at least two first cameras; then, the second three-dimensional coordinates of the target object being gazed at by the person, that is, the ending point of the line-of-sight vector, are determined by means of the image data collected by the at least one second camera; and finally, the line-of-sight vector of the driver is accurately determined by means of the first three-dimensional coordinates and the second three-dimensional coordinates. In this way, the starting point and ending point positions of the line-of-sight vector determined are accurately determined, thereby ensuring the accurate determination of the line-of-sight vector. Moreover, the line-of-sight end point recognized by the image recognition technology does not have to rely on the position of a preset LED light, but the three-dimensional coordinates of the target object being gazed at by the person are determined by means of the image recognition technology of the camera, thereby being able to overcome the limitations in the prior art and being able to determine the line-of-sight vector of the person with fewer restrictions.

[0032] The working principle and the process during specific implementation of the method for determining the line-of-sight vector of a person in a vehicle according to the present disclosure will be described below with reference to the accompanying drawings. Among them, Figure 1 FIG. shows a flowchart of a method 100 for determining the line-of-sight vector of a person in a vehicle according to an embodiment of the present disclosure, Figure 2 FIG. shows a schematic diagram of a device 200 for determining the line-of-sight vector of a person in a vehicle according to an embodiment of the present disclosure, and Figure 3 FIG. shows a schematic diagram of a device 300 for determining the line-of-sight vector of a person in a vehicle according to another embodiment of the present disclosure.

[0033] From Figure 1It can be seen that the vehicle configured based on the method 100 for determining the line-of-sight vector of a person in a vehicle disclosed in the present disclosure has multiple cameras. After the posture of the person is determined, at least two first cameras among the multiple cameras can capture the pupil positions of the person, and at least one second camera among the multiple cameras can capture the target object being gazed at by the person. In other words, for example, after the posture of the driver is determined, at least two cameras can capture the pupils of the driver to determine the three-dimensional coordinates of the left and right pupils of the driver. Correspondingly, for the target object being gazed at, such as a dedicated measuring tool, at least one camera can also capture the target object being gazed at by the person, so as to determine the three-dimensional coordinates of the target object being gazed at. Here, in order to capture the pupils of the driver, the first camera is, for example, arranged on this side of the front windshield, such as at the instrument panel or the center console screen, facing the interior of the vehicle, that is, in the direction of the driver. Of course, those skilled in the art should understand that it can also be arranged, for example, above the front of the driver, such as at the sun visor, as long as it can capture the pupil positions of the driver. Since the second camera needs to capture the position of the target object being gazed at by the driver, and the target object is usually in front of the driver, such as in front of the driver's cab, the camera can be arranged, for example, at the skylight or the edge of the skylight at this time, as long as it can capture the target object. If a camera can capture both the pupils of the driver and the target object being gazed at by the driver, then the first camera and the second camera can also be reused. That is to say, preferably, in an implementation form according to the present disclosure, the first camera and the second camera are reused with each other.

[0034] Here, the method 100 for determining the line-of-sight vector of a person in a vehicle according to the present disclosure includes at least the following three steps, namely: First, in method step 110, the first three-dimensional coordinates of at least one pupil of the person, that is, the three-dimensional coordinates of the starting point of the line of sight, will be determined based on the image data collected by the at least two first cameras; then, in method step 120, the second three-dimensional coordinates of the target object being gazed at by the person, that is, the three-dimensional coordinates of the end point of the line of sight, will be determined based on the image data collected by the at least one second camera; and finally, in method step 130, the specific value of the line-of-sight vector will be determined based on the at least two three-dimensional coordinates obtained in method steps 110 and 120, that is, in method step 130, the line-of-sight vector of the driver will be determined based on the first three-dimensional coordinates and the second three-dimensional coordinates. Here, in the method 100 for determining the line-of-sight vector of a person in a vehicle according to the present disclosure, first, the first three-dimensional coordinates of at least one pupil of the person, that is, the starting point of the line-of-sight vector, are determined by means of the image data collected by the at least two first cameras; then, the second three-dimensional coordinates of the target object being gazed at by the person, that is, the end point of the line-of-sight vector, are determined by means of the image data collected by the at least one second camera; and finally, the line-of-sight vector of the driver is accurately determined by means of the first three-dimensional coordinates and the second three-dimensional coordinates. In this way, the starting and ending positions of the line-of-sight vector determined are accurately determined, thus ensuring the accurate determination of the line-of-sight vector. Moreover, the end point of the line of sight recognized by the image recognition technology does not have to rely on the position of a preset LED light, but rather the three-dimensional coordinates of the target object being gazed at by the person are determined by means of the image recognition technology of the camera, thereby being able to overcome the limitations in the prior art and enabling the determination of the line-of-sight vector of the person with fewer restrictions.

[0035] In order to facilitate the processing of the first three-dimensional coordinates and the second three-dimensional coordinates, it is preferable that these cameras are initially in the same coordinate system. That is to say, preferably, in an implementation form according to the present disclosure, the method 100 for determining the line-of-sight vector of a person in a vehicle according to the present disclosure further includes (not shown in the figure): calibrating the multiple cameras (such as the above-mentioned first camera and second camera) in the same coordinate system. More preferably, in the method 100 for determining the line-of-sight vector of a person in a vehicle shown in an implementation form according to the present disclosure, calibrating the multiple cameras in the same coordinate system includes: calibrating the multiple cameras in the same coordinate system by means of a black-and-white checkerboard. Specifically, during operation, first, for example, calibrate the first camera in a coordinate system by means of a black-and-white checkerboard, and then use the same black-and-white checkerboard to calibrate the second camera in the same coordinate system. In this way, the three-dimensional coordinates of the pupil obtained by the first camera and the three-dimensional coordinates of the target object gazed at by the person in the vehicle obtained by the second camera will be in the same coordinate system, so that addition and subtraction processing can be performed without additional conversion and the line vector can be easily obtained. Here, optionally, in an implementation form according to the present disclosure, the person includes the driver or passenger of the vehicle.

[0036] Preferably, in an implementation form according to the present disclosure, the target object is configured as a measurement tool including a plurality of calibration points, and wherein the second three-dimensional coordinates are determined as the three-dimensional coordinates at the end of the measurement tool. Specifically, in implementation, the measurement tool is, for example, configured as a measurement tool with the code name V01, which has, for example, four to five positioning balls. During specific operation, it can, for example, be fixed to the vehicle body with the help of a support frame and placed at a desired position. When the driver gazes at the end of the measurement tool, the three-dimensional coordinates of each positioning ball can be determined by means of a camera. Then, since the structure of the measurement tool is fixed and is also known to the camera used to determine its end, the three-dimensional coordinates of the end of the measurement tool, that is, the three-dimensional coordinates of the target object gazed at by the driver, can be calculated by means of the three-dimensional coordinates of these positioning balls. For this purpose, more preferably, in an implementation form according to the present disclosure, the method further includes: fixing the measurement tool to a preset position by means of a fixing tool. Thereby, the measurement tool can be fixed, facilitating the subsequent determination of the line-of-sight vector. Generally speaking, in an implementation form according to the present disclosure, determining the second three-dimensional coordinates of the target object gazed at by the person based on the image data collected by the at least one second camera includes: determining a plurality of third three-dimensional coordinates of the plurality of calibration points based on the image data collected by the at least one second camera; and determining the second three-dimensional coordinates based on the physical structure data of the measurement tool and the third three-dimensional coordinates.

[0037] In addition to being implemented by means of software control methods and the like, the above technical solution can also be implemented, for example, through corresponding hardware circuits. In other words, the above method 100 for determining the line-of-sight vector of a person in a vehicle can be implemented by software stored in a computer-readable storage medium in combination with corresponding hardware components. Computer-readable program instructions for executing various embodiments of the present disclosure are uploaded on the computer-readable storage medium. The computer-readable storage medium can be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in a groove storing instructions thereon, and any suitable combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0038] For this reason, a second aspect of the present disclosure proposes a device for determining the line-of-sight vector of a person in a vehicle. Figure 2 The schematic diagram of a device 200 for determining the line-of-sight vector of a person in a vehicle according to an embodiment of the present disclosure is shown. The vehicle is configured with a plurality of cameras, and after the posture of the person is determined, at least two first cameras among the plurality of cameras can capture the pupil positions of the person, and at least one second camera among the plurality of cameras can capture the target object gazed at by the person. As can be seen from Figure 2 this, the device 200 includes a memory 220 and a processor 210. Computer-readable instructions are stored on the memory 220. When the computer-readable instructions are executed by the processor 210, the processor 210 is caused to implement the method 100 for determining the line-of-sight vector of a person in a vehicle proposed in the first aspect of the present disclosure.

[0039] Furthermore, a third aspect of the present disclosure provides a computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the processor is caused to implement the method for determining a line-of-sight vector of a person in a vehicle according to the first aspect of the present disclosure. Figure 3 FIG. shows a schematic diagram of an apparatus 300 for determining a line-of-sight vector of a person in a vehicle according to another embodiment of the present disclosure. As can be seen from Figure 3 it, it should be understood that the apparatus 300 for determining a line-of-sight vector of a person in a vehicle can be implemented to implement the Figure 1 function of the method 100 for determining a line-of-sight vector of a person in a vehicle in. As can be seen from Figure 3 it, the apparatus 300 for determining a line-of-sight vector of a person in a vehicle includes a central processing unit (CPU) 301 (such as a processor), which can execute various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 302 or computer program instructions loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the apparatus 300 can also be stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 404. An input / output (I / O) interface 305 is also connected to the bus 304.

[0040] A plurality of components in the apparatus 300 for determining a line-of-sight vector of a person in a vehicle are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a magnetic disk, an optical disc, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the apparatus 300 for determining a line-of-sight vector of a person in a vehicle to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0041] The various methods described above, such as method 100 for determining the line-of-sight vector of a person in a vehicle, can be executed by the processing unit 301. For example, in some embodiments, method 100 for determining the line-of-sight vector of a person in a vehicle can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed onto the apparatus 300 for determining the line-of-sight vector of a person in a vehicle via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by CPU 301, one or more actions or steps of method 100 for determining the line-of-sight vector of a person in a vehicle described above can be executed, that is: determining the first three-dimensional coordinates of at least one pupil of the person based on the image data collected by the at least two first cameras; determining the second three-dimensional coordinates of the target object gazed at by the person based on the image data collected by the at least one second camera; and determining the line-of-sight vector of the driver based on the first three-dimensional coordinates and the second three-dimensional coordinates. Among the methods for determining the line-of-sight vector of a person in a vehicle according to the present disclosure, first, the first three-dimensional coordinates of at least one pupil of the person, that is, the starting point of the line-of-sight vector, are determined by means of the image data collected by the at least two first cameras; then, the second three-dimensional coordinates of the target object gazed at by the person, that is, the end point of the line-of-sight vector, are determined by means of the image data collected by the at least one second camera; and finally, the line-of-sight vector of the driver is accurately determined by means of the first three-dimensional coordinates and the second three-dimensional coordinates. In this way, the starting and ending positions of the line-of-sight vector determined in this way can be accurately determined, so as to ensure the accurate determination of the line-of-sight vector. Furthermore, the line-of-sight end point recognized by the image recognition technology does not have to rely on the position of a preset LED light, but the three-dimensional coordinates of the target object gazed at by the person are determined by means of the image recognition technology of the camera, so that the limitations in the prior art can be eliminated, and the determination of the line-of-sight vector of the person can be carried out with fewer restrictions.

[0042] Preferably, in an implementation form according to the present disclosure, when the computer program is loaded into RAM 303 and executed by CPU 301, one or more actions or steps of method 100 for determining the line-of-sight vector of a person in a vehicle described above can be executed, that is: calibrating the multiple cameras in the same coordinate system. More preferably, in an implementation form according to the present disclosure, calibrating the multiple cameras in the same coordinate system includes: calibrating the multiple cameras in the same coordinate system by means of a black and white checkerboard.

[0043] Optionally, in an implementation form according to the present disclosure, the person includes the driver or passenger of the vehicle. Preferably, in an implementation form according to the present disclosure, the target object is configured as a measurement tool including a plurality of calibration points, and wherein the second three-dimensional coordinates are determined as the three-dimensional coordinates at the end of the measurement tool. More preferably, in an implementation form according to the present disclosure, when the computer program is loaded into the RAM 303 and executed by the CPU 301, one or more actions or steps in the method 100 for determining the line-of-sight vector of the person in the vehicle described above can be executed, that is: fixing the measurement tool to a preset position by means of a fixing tool. In an implementation form according to the present disclosure, determining the second three-dimensional coordinates of the target object gazed at by the person based on the image data collected by the at least one second camera includes: determining a plurality of third three-dimensional coordinates of the plurality of calibration points based on the image data collected by the at least one second camera; and determining the second three-dimensional coordinates based on the physical structure data of the measurement tool and the third three-dimensional coordinates. Preferably, in an implementation form according to the present disclosure, the first camera and the second camera are multiplexed with each other.

[0044] In summary, in the method for determining the line-of-sight vector of the person in the vehicle according to the present disclosure, first, the first three-dimensional coordinates of at least one pupil of the person are determined by means of the image data collected by the at least two first cameras, that is, the starting point of the line-of-sight vector; then, the second three-dimensional coordinates of the target object gazed at by the person are determined by means of the image data collected by the at least one second camera, that is, the end point of the line-of-sight vector; finally, the line-of-sight vector of the driver is accurately determined by means of the first three-dimensional coordinates and the second three-dimensional coordinates. In this way, the starting point and end point positions of the line-of-sight vector determined are accurately determined, so as to ensure the accurate determination of the line-of-sight vector. Moreover, the line-of-sight end point recognized by the image recognition technology does not have to rely on the position of the preset LED lamp, but determines the three-dimensional coordinates of the target object gazed at by the person by means of the image recognition technology of the camera, so that the limitations in the prior art can be eliminated, and the determination of the line-of-sight vector of the person can be carried out with fewer restrictions.

[0045] The main improvement of the above-mentioned new fixation reference system setting lies in allowing the fixation target point, i.e., the end of the measuring tool, to move freely in three-dimensional space and automatically obtaining its three-dimensional position in three-dimensional space, i.e., three-dimensional coordinates. Therefore, the method 100 for determining the line-of-sight vector of a person in a vehicle proposed according to the present disclosure eliminates the work of manually calibrating the three-dimensional position of a point in three-dimensional space and allows for a greater degree of freedom of movement of the driver's fixation target. In addition, the position of the driver's face (e.g., the position of the pupil) will not be measured and tracked by the SmartTrack device, but by a multi-camera system. This system does not require the test subject to wear any hardware device and has almost no adverse effects.

[0046] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, in any case, the embodiments should be regarded as exemplary and non-limiting. In addition, it is obvious that the word "comprising" does not exclude other elements and steps, and the word "a" does not exclude a plurality. A plurality of elements recited in the apparatus claims can also be implemented by one element. The terms first, second, etc. are used to denote names and do not denote any particular order.

Claims

1. A method for determining a line-of-sight vector of a person inside a vehicle, characterized in that, The vehicle is equipped with multiple cameras, and after the posture of the person is determined, at least two first cameras among the multiple cameras can capture the pupil positions of the person, and at least one second camera among the multiple cameras can capture the target object being gazed at by the person. The method includes: Determining first three-dimensional coordinates of at least one pupil of the person based on the image data collected by the at least two first cameras; Determining second three-dimensional coordinates of the target object being gazed at by the person based on the image data collected by the at least one second camera; and Determining the line-of-sight vector of the driver based on the first three-dimensional coordinates and the second three-dimensional coordinates.

2. The method according to claim 1, wherein The method further includes: Calibrating the multiple cameras in the same coordinate system.

3. The method according to claim 2, characterized in that, Calibrating the multiple cameras in the same coordinate system includes: Calibrating the multiple cameras in the same coordinate system by means of a black-and-white checkerboard.

4. The method according to claim 1, wherein The person includes the driver or a passenger of the vehicle.

5. The method according to any one of claims 1 to 4, characterized in that The target object is configured to be a measuring tool including multiple calibration points, and wherein, the second three-dimensional coordinates are determined as the three-dimensional coordinates at the end of the measuring tool.

6. The method according to claim 5, wherein The method further includes: Fixing the measuring tool to a preset position by means of a fixing tool.

7. The method according to claim 5, wherein Determining second three-dimensional coordinates of the target object being gazed at by the person based on the image data collected by the at least one second camera includes: Determining multiple third three-dimensional coordinates of the multiple calibration points based on the image data collected by the at least one second camera; and Determining the second three-dimensional coordinates based on the physical structure data of the measuring tool and the third three-dimensional coordinates.

8. The method according to claim 1, characterized in that, The first camera and the second camera are multiplexed with each other.

9. A device for determining a line-of-sight vector of a person inside a vehicle, characterized in that, The vehicle is equipped with multiple cameras, and after the posture of the person is determined, at least two first cameras among the multiple cameras can capture the pupil positions of the person, and at least one second camera among the multiple cameras can capture the target object being gazed at by the person. The apparatus includes a memory and a processor. Computer-readable instructions are stored on the memory. When the computer-readable instructions are executed by the processor, the processor is caused to implement the method for determining the line-of-sight vector of a person in a vehicle according to any one of claims 1 to 8.

10. A computer-readable storage medium, wherein, Computer-readable instructions are stored on the computer-readable storage medium. When the computer-readable instructions are executed by a processor, the processor is caused to implement the method for determining the line-of-sight vector of a person in a vehicle according to any one of claims 1 to 8.

Images