An air projection control method and system based on eye tracking

Abstract

The application provides an air projection control method and system based on eyeball tracking, and the method comprises the following steps: detecting whether there is an air projection operation instruction, wherein the air projection operation instruction comprises voice request information, vehicle state information and / or eyeball label information; if there is no air projection operation instruction, the imaging surface of the air projection is in a preset position; if there is an air projection operation instruction, based on a response priority rule, the angle and the imaging area of the air projection are adjusted according to the voice request information, the vehicle state information and / or the eyeball label information. The application can comprehensively combine voice request, vehicle state and eyeball tracking technology, adaptively adjust the angle and the projection size of projection imaging according to a preset response priority, meet the projection display requirements of personnel on each seat in the vehicle, enhance the functionality and interestingness of the vehicle-mounted projection function, and improve the experience of the driver and the passenger.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent cockpit, more particularly, to an air projection control method and system based on eyeball tracking. BACKGROUND

[0002] With the development and popularization of automobiles, in addition to the basic function of meeting the safe driving of vehicles, the requirements for the operation experience and comprehensive functionality of vehicles are increasingly high, and more and more functional designs tend to be intelligent, so that vehicles can better serve the drivers and passengers and improve the comprehensive performance of vehicles.

[0003] For the intelligent display system used in vehicles, there are several main methods as follows:

[0004] 1. A head-up display device can be installed in, for example, an instrument panel of a vehicle, and a position of a pupil of a driver is continuously monitored by an eye tracker and provided to a controller, and the controller provides a video including state information of the vehicle, driving information of the vehicle, navigation information, a front environment of the vehicle, or a rear environment of the vehicle, etc. to the driver in response to the position of the pupil of the driver input from the eye tracker. In this scheme, the eye tracker is used to track the pupil of the driver, and the vehicle-related information is displayed through the head-up display device. This scheme only displays information to the driver and does not consider the information reading needs of the passengers in other seats, which has limitations.

[0005] 2. A high-definition camera captures high-definition images of a face of a driver at a certain frequency, and transmits image information to a single-chip microcomputer, the single-chip microcomputer processes the photos captured by the high-definition camera, and uses an intelligent image processing algorithm to determine a face orientation and a gaze direction of the eye; then, the single-chip microcomputer controls a rotating motor to rotate according to the rotation angle of the rotating motor calculated based on the face orientation and the gaze direction of the eye; finally, the single-chip microcomputer receives a vehicle speed of a center console of the vehicle, and controls a miniature laser projector to project the vehicle speed onto a windshield of the vehicle for the driver to observe. In this scheme, the angle of the miniature laser projector can be controlled by the rotating motor, so that the vehicle speed information is projected onto the windshield of the vehicle for the driver to observe. However, in this scheme, the gaze direction of the eye of the driver is still used as the basis for projection adjustment, and the information reading needs of the passengers in other seats are not considered. Moreover, since the information display is projected onto the windshield of the vehicle, the information display is a planar display, the imaging plane is fixed, and only the imaging position and the imaging size can be adjusted, which also has limitations.

[0006] Air projection, also known as air imaging, aerial stereoscopic imaging, fog screen imaging, etc., is a new projection technology. The laser is imaged in the air without any projection screen. Meanwhile, the function in the screen can be directly operated in the air, which feels like being in the future world in the movie. At present, there are cases of using air projection technology in the in-vehicle display function. However, the display area is still the field of view of the driver. Although the operation experience of the driver can be greatly improved, the projection area and the projection visual angle are small for the other passengers, and the user experience needs to be improved. SUMMARY

[0007] The present application aims at the technical problems in the prior art, and provides an air projection control method and system based on eye tracking. The air projection control method and system based on eye tracking can comprehensively combine voice request, vehicle state and eye tracking technology, adaptively adjust the angle and size of projection imaging according to the preset response priority, meet the projection display needs of the passengers in each seat in the vehicle, enhance the functionality and interest of the vehicle projection function, and improve the experience of the driver and passenger.

[0008] According to a first aspect of the present application, an air projection control method based on eye tracking is provided, comprising:

[0009] detecting whether there is an air projection operation instruction, the air projection operation instruction including voice request information, vehicle state information and / or eye tracking information;

[0010] if there is no air projection operation instruction, the imaging surface of the air projection is in a preset position;

[0011] if there is an air projection operation instruction, the angle and imaging area of the air projection are adjusted based on the response priority rule according to the voice request information, vehicle state information and / or eye tracking information.

[0012] On the basis of the above technical solution, the present application can also be improved as follows.

[0013] Optionally, the voice request information is detected, comprising:

[0014] acquiring a voice wake-up signal, and acquiring voice request information for starting the air projection function according to the voice wake-up signal;

[0015] identifying the microphone array sound area where the voice wake-up signal is located, and determining the passenger seat that sends the voice request according to the sound area identification result.

[0016] Optionally, the vehicle state information at least includes: vehicle driving mode information and vehicle speed, and the driving mode information includes active driving mode or auxiliary driving mode.

[0017] Optionally, detect eye tag information, including:

[0018] Acquire real-time video sequences inside the vehicle, track facial information in the real-time video sequences, and identify whether eye features are present in the facial information;

[0019] When eye features are identified, the human eye is illuminated with low-power active near-infrared light and a sensor array is used to capture near-infrared images.

[0020] Identify eye features in near-infrared images, estimate the gaze direction and gaze point of the corresponding eyeball based on the eye features, compare the angle 'a' between the air projection position, the eye, and the horizontal line with the angle 'b' between the gaze direction and the horizontal line, and determine that the corresponding eyeball is gazing at the air projection position when the difference between angle 'a' and angle 'b' is less than the error threshold.

[0021] Based on the distance from the gaze point to the corresponding eyeball, the position of the eyeball that is gazing at the air projection position is matched with the vehicle seat, and eyeball tags are assigned to all eyes gazing at the air projection position according to the corresponding seat attributes.

[0022] Optionally, the response priority rules include:

[0023] Voice request priority > vehicle status priority > eye tag priority.

[0024] Optionally, the preset position is such that the air projection position faces the driver's seat, and the imaging area is located in the calibrated position most suitable for the driver's eye to view from the driver's seat.

[0025] Optionally, adjusting the angle and imaging area of ​​the air projection based on the response priority rules, according to the voice request information, vehicle status information, and / or eye tag information, includes:

[0026] Obtain the standard straight-line distance A, standard vertical height difference B, and standard lateral distance C between the imaging area of ​​the air projection and each seat;

[0027] To determine the type of the current air projection operation command, consider the following principle: voice request priority > vehicle status priority > eye tag priority.

[0028] If a voice request is currently in effect, the passenger seat at the location of the sound source is determined based on the voice request information, and the eye tag priority of the passenger in that seat is set to the highest.

[0029] If there is no current voice request, the driving mode information and the vehicle speed of the vehicle are obtained according to the vehicle state information, when the driving mode information is the active driving mode, the eye label of the driver is shielded; when the driving mode information is the auxiliary driving mode, the priority of the eye label of the driver is set after the priority of the eye label of the remaining seats; when the vehicle speed is 0, the priority of the eye label of the driver is set before the priority of the eye label of the remaining seats.

[0030] If the eye of the remaining seat at the multiple gaze air projection positions is tracked at present, the eye label with the highest priority is taken as the target eye label; wherein the priority of the eye label of the remaining seats except the main driving position is ranked as: copilot > right rear > left rear > middle rear;

[0031] The straight line distance A1 from the corresponding eye to the imaging position is calculated according to the target eye label, the straight line distance A1 is compared with the standard straight line distance A, and the magnification or reduction of the imaging area is adjusted according to the comparison result of the straight line distance.

[0032] The vertical height difference B1 from the corresponding eye to the imaging position is calculated according to the target eye label, the vertical height difference B1 is compared with the standard vertical height difference B, and the longitudinal inclination angle of the imaging area is adjusted according to the comparison result of the vertical height difference.

[0033] The lateral distance C1 from the corresponding eye to the imaging position is calculated according to the target eye label, the lateral distance C1 is compared with the standard lateral distance C, and the lateral inclination angle of the imaging area is adjusted according to the comparison result of the lateral distance.

[0034] According to the second aspect of the present application, an air projection control system based on eye tracking is provided, comprising:

[0035] The acquisition module is used for detecting whether there is an air projection operation instruction, the air projection operation instruction includes voice request information, vehicle state information and / or eye label information;

[0036] The adjustment module is used for if it is determined that there is no air projection operation instruction, controlling the imaging surface of the air projection to be at a preset position;

[0037] And if it is determined that there is an air projection operation instruction, adjusting the angle and the imaging area of the air projection based on a response priority rule according to the voice request information, the vehicle state information and / or the eye label information.

[0038] According to the third aspect of the present application, an electronic device is provided, comprising a memory and a processor, the processor is used for executing the computer management type program stored in the memory to realize the steps of the above-mentioned air projection control method based on eye tracking.

[0039] According to a fourth aspect of the present application, a computer readable storage medium is provided, having stored thereon a computer management program which, when executed by a processor, implements the steps of the above-mentioned air projection control method based on eye tracking.

[0040] The air projection control method, system, electronic device and storage medium based on eye tracking provided by the present application can comprehensively combine voice requests, vehicle states and eye tracking technology, consider the projection display requirements of personnel on each seat in the vehicle, adaptively adjust the angle and size of projection imaging according to a preset response priority, enhance the functionality and interest of the vehicle projection function, and improve the experience of the driver and passenger. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of the air projection control method based on eye tracking provided by the present application is shown in FIG. 1.

[0042] Figure 2 A functional module diagram of the method involved in an embodiment of the present application is shown in FIG. 2.

[0043] Figure 3 A diagram showing adjustment of the imaging angle according to the sound source in an embodiment of the present application is shown in FIG. 3.

[0044] Figure 4 A diagram showing the included angle a and the included angle b in an embodiment of the present application is shown in FIG. 4.

[0045] Figure 5 A diagram showing angle and imaging size adjustment through eye tracking in an embodiment of the present application is shown in FIG. 5.

[0046] Figure 6 A block diagram of the air projection control system based on eye tracking provided by the present application is shown in FIG. 6.

[0047] Figure 7 A hardware structure diagram of a possible electronic device provided by the present application is shown in FIG. 7.

[0048] Figure 8 A hardware structure diagram of a possible computer readable storage medium provided by the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0049] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0050] Figure 1 A flowchart of the air projection control method based on eye tracking provided by the present application is shown in FIG. 1. Figure 2 A functional module diagram of the method involved in an embodiment of the present application is shown in FIG. 2. Figure 1 andFigure 2 As shown in the method comprises:

[0051] Detecting whether there is an air projection operation instruction, the air projection operation instruction comprising voice request information, vehicle state information and / or eye label information;

[0052] If there is no air projection operation instruction, the imaging plane of the air projection is in a preset position;

[0053] If there is an air projection operation instruction, based on a response priority rule, the angle and imaging area of the air projection are adjusted according to the voice request information, vehicle state information and / or eye label information.

[0054] It can be understood that based on the defects in the background art, the embodiment of the present application proposes an air projection control method based on eye tracking. Among them, the voice request information can be collected by the microphone dispersedly arranged near each seat, the vehicle state can be collected through the communication of the bus and the vehicle TBOX, and the eye label information can be collected based on the camera through image recognition. The method comprehensively combines the voice request information of the personnel on each seat, the current vehicle state and the eye tracking technology, considers the projection display requirements of the personnel on each seat in the vehicle, adaptively adjusts the angle and size of the projection imaging according to the preset response priority, enhances the functionality and interest of the vehicle-mounted projection function, and improves the experience of the driver and passenger.

[0055] In a possible embodiment, the voice request information is detected, comprising:

[0056] Acquiring a voice wake-up signal, and acquiring voice request information for starting the air projection function according to the voice wake-up signal;

[0057] Identifying the microphone array sound area where the voice wake-up signal is located, and determining the passenger seat that sends the voice request according to the sound area identification result.

[0058] It can be understood that the microphone array is arranged at the position adjacent to each seat in the vehicle to collect the voice information of the driver and passenger on each seat. Through the sound field of the collected voice, the position where the voice is sent can be determined, so as to determine the passenger seat that sends the voice demand. Figure 3 As shown, the projection angle of the rotating air projection device can also be adjusted according to the passenger seat obtained by the voice request information, so as to adjust the image plane angle of the air projection, so that it is better matched with the passenger's visual field of sending the voice demand.

[0059] In a possible embodiment, the vehicle state information at least includes: the driving mode information and the vehicle speed of the vehicle, and the driving mode information includes the active driving mode or the auxiliary driving mode.

[0060] It can be understood that whether the vehicle is in a driving state can be determined by whether the vehicle speed is 0, and whether the current is in an active driving mode or an auxiliary driving mode can be determined by the driving mode information. The vehicle non-driving state, the active driving state and the auxiliary driving state obtained by the embodiment can be used as a basis for determining the priority of the eye label of the person in the main driver seat in the subsequent steps.

[0061] In a possible embodiment, the eye label information is detected, including:

[0062] An in-vehicle real-time video sequence is obtained, and face information in the real-time video sequence is tracked, and whether the eye feature exists in the face information is identified;

[0063] When the eye feature is identified, the eye is irradiated by low-power active near-infrared, and a near-infrared image is captured by using a sensing array;

[0064] The eye feature in the near-infrared image is identified, and the gaze direction and the gaze point of the corresponding eye are estimated according to the eye feature, as shown in Figure 4 The angle a between the air projection position-eye-horizontal line is compared with the angle b between the gaze direction-horizontal line, and when the difference between the angle a and the angle b is less than an error threshold, it is determined that the corresponding eye is gazing at the air projection position. For example, the error threshold range can be set to ±10%, if (a-b) / b∈[-10%,+10%], it is determined that the eye is gazing at the air projection position;

[0065] According to the distance from the gaze point to the corresponding eye, the position of the eye gazing at the air projection position is matched with the vehicle seat, and all the eyes gazing at the air projection position are labeled with eye labels according to the corresponding seat attributes.

[0066] It can be understood that through the steps of the embodiment, the eyes gazing at the air projection position on all the faces in the vehicle can be identified, and the eye labels are added to the corresponding eyes according to the preset seat numbers, which are used for identity recognition of the detected eyes. For example, if multiple eyes are tracked, each eye is labeled with an eye label according to the seat it is in, such as the main driver, the co-driver, the second-row left, the second-row middle, the second-row right, etc.

[0067] In a possible embodiment, the response priority rule includes:

[0068] Voice request priority> vehicle state priority> eye label priority.

[0069] It can be understood that, for example, first, the position of the air projection demand is judged according to the voice request information, and then the imaging plane angle of the air projection is adjusted for the position, so that the imaging plane is directed to the passenger who issues the voice request; if the demand position cannot be well judged through the voice request information, the current vehicle state is judged according to the collected vehicle state information, and the final air projection adjustment scheme is determined in combination with the current vehicle state type and the collected eye label.

[0070] In a possible embodiment, the preset position is an air projection position directed to a main driver seat, and an imaging area is at a position most suitable for eye viewing of the main driver seat. The angle, distance parameter and standard sitting posture parameter (line of sight height, line of sight angle and line of sight distance) correspond to each other.

[0071] It can be understood that, in terms of use frequency, the main driver position uses the air projection most frequently, so that the position most suitable for eye viewing of the main driver seat is calibrated in advance as an initial position (preset position) of the air projection, so as to reduce the adjustment times of the air projection.

[0072] In a possible embodiment, the angle and imaging area of the air projection are adjusted according to the voice request information, vehicle state information and / or eye label information based on a response priority rule, and the adjusting includes:

[0073] A standard straight line distance A, a standard vertical height difference B and a standard lateral distance C between an imaging area of the air projection and each seat are acquired;

[0074] The type of the current air projection operation instruction is judged, and the principle of voice request priority> vehicle state priority> eye label priority is combined:

[0075] If there is a voice request at present, the passenger seat of the sound source position is determined according to the voice request information, and the eye label priority of the passenger in this seat is set to the highest;

[0076] If there is no voice request at present, the driving mode information and vehicle speed of the vehicle are acquired according to the vehicle state information, when the driving mode information is a main driving mode, the eye label of the driver is shielded; when the driving mode information is an auxiliary driving mode, the eye label priority of the driver is set to be behind the eye label priorities of the remaining seats; when the vehicle speed is 0, the eye label priority of the driver is set to be ahead of the eye label priorities of the remaining seats;

[0077] If the eyes of the remaining seats looking at the air projection position are tracked at present, the eye label with the highest priority is taken as a target eye label; wherein the eye label priority of the remaining seats except the main driver seat is sorted as: co-driver> right rear> left rear> middle rear;

[0078] According to the target eyeball label, the straight line distance A1 of the corresponding eyeball to the imaging position is calculated, the straight line distance A1 is compared with the standard straight line distance A, and the magnification or reduction of the imaging area is adjusted according to the comparison result of the straight line distance;

[0079] According to the target eyeball label, the vertical height difference B1 of the corresponding eyeball to the imaging position is calculated, the vertical height difference B1 is compared with the standard vertical height difference B, and the longitudinal inclination angle of the imaging area is adjusted according to the comparison result of the vertical height difference;

[0080] According to the target eyeball label, the horizontal distance C1 of the corresponding eyeball to the imaging position is calculated, the horizontal distance C1 is compared with the standard horizontal distance C, and the horizontal inclination angle of the imaging area is adjusted according to the comparison result of the horizontal distance.

[0081] It can be understood that, as Figure 5 The angle and imaging size adjustment diagram through eyeball tracking in the embodiment is shown. According to the imaging principle of near large and far small, the standard straight line distance A is used to judge whether the area size of the imaging picture is suitable for the detected target eyeball. When the seat is close to the air projection device (for example, the driver or the front passenger seat), the imaging area watched by the eyes is larger; when the seat is far away from the air projection device (for example, the rear seat), the imaging area watched by the eyes is smaller, so the straight line distance A1 of the corresponding eyeball to the imaging position is compared with the standard straight line distance A, so as to adjust the area size of the imaging picture, so that it is more suitable for the corresponding eyeball to watch. Figure 5 In the embodiment, it is assumed that the straight line distance A1 between the eyeball and the imaging picture is 60 cm, and it is determined that the corresponding eyeball and the imaging picture are at a more suitable distance, and the area size of the imaging picture can not be changed. Changing the area size of the imaging picture is to scale the imaging picture, which is a conventional technical means of projection technology, which will not be described here.

[0082] Similarly, the vertical height difference B1 of the corresponding eyeball to the imaging position can be used to judge whether the longitudinal inclination angle of the imaging area, that is, the pitch angle of the image plane, needs to be adjusted. As Figure 5As shown, the vertical height difference B1 can be obtained by subtracting the height of the top of the image from the bottom by the height H of the eyeball relative to the bottom of the image. This vertical height difference B1 is related to the height h of the negative refractive glass. By rotating the longitudinal angle of the negative refractive glass, the height of the top of the image from the bottom can be adjusted, thereby adjusting the vertical height difference B1 from the corresponding eyeball to the imaging position and the longitudinal tilt angle (i.e., the vertical tilt angle) of the imaging area. The lateral distance C1 from the corresponding eyeball to the imaging position reflects the lateral tilt angle of the image, such as the left-right tilt angle. By adjusting the vertical height difference B1 and / or the lateral distance C1 from the corresponding eyeball to the imaging position using the adjustment mechanism, the size of the image can be adjusted to present the image to the corresponding eyeball at a better angle and a more suitable size, improving the user experience of the air projection at each seat.

[0083] Figure 6 A structural diagram of an eye-tracking-based air projection control system is provided for an embodiment of the present invention, as shown below. Figure 6 As shown, an eye-tracking-based air projection control system includes an acquisition module and an adjustment module, wherein:

[0084] The acquisition module is used to detect whether there is an air projection operation command. The air projection operation command includes voice request information, vehicle status information and / or eye tag information. For example, it acquires voice request information from each seat through a microphone array, acquires vehicle status information through communication between the bus and TBOX, performs in-vehicle face eye recognition through a camera, and obtains eye tag information through the recognized eye.

[0085] The adjustment module is used to control the imaging surface of the air projection to a preset position if it is determined that there is no air projection operation command.

[0086] It is also used to adjust the angle and imaging area of ​​the air projection based on the voice request information, vehicle status information and / or eye tag information, according to the response priority rules if an air projection operation command is determined to exist.

[0087] It is understood that the eye-tracking-based air projection control system provided by the present invention corresponds to the eye-tracking-based air projection control method provided in the foregoing embodiments. The relevant technical features of the eye-tracking-based air projection control system can be referred to the relevant technical features of the eye-tracking-based air projection control method, and will not be repeated here.

[0088] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 7As shown, an embodiment of the present invention provides an electronic device 700, including a memory 710, a processor 720, and a computer program 711 stored in the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 711, it performs the following steps:

[0089] Detect the presence of an air projection operation command, which includes voice request information, vehicle status information, and / or eye tag information;

[0090] If no air projection operation command is given, the imaging surface of the air projection will be in the preset position.

[0091] If an air projection operation command is given, the angle and imaging area of ​​the air projection are adjusted based on the response priority rules, the voice request information, vehicle status information, and / or eye tag information.

[0092] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 8 As shown, this embodiment provides a computer-readable storage medium 800, on which a computer program 811 is stored. When the computer program 811 is executed by a processor, it performs the following steps:

[0093] Detect the presence of an air projection operation command, which includes voice request information, vehicle status information, and / or eye tag information;

[0094] If no air projection operation command is given, the imaging surface of the air projection will be in the preset position.

[0095] If an air projection operation command is given, the angle and imaging area of ​​the air projection are adjusted based on the response priority rules, the voice request information, vehicle status information, and / or eye tag information.

[0096] This invention provides an eye-tracking-based air projection control method, system, and storage medium that can comprehensively combine voice requests, vehicle status, and eye-tracking technology. It considers the projection display needs of people in each seat in the vehicle and adaptively adjusts the projection angle and size according to preset response priorities, thereby enhancing the functionality and fun of in-vehicle projection and improving the experience of drivers and passengers.

[0097] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0103] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for controlling air projection based on eye tracking, characterized in that, include: Detecting the presence of an air projection operation command, wherein the air projection operation command includes voice request information, vehicle status information, and / or eye tag information; wherein, detecting eye tag information includes: Acquire real-time video sequences inside the vehicle, track facial information in the real-time video sequences, and identify whether eye features are present in the facial information; When eye features are identified, the human eye is illuminated with low-power active near-infrared light and a sensor array is used to capture near-infrared images. Identify eye features in near-infrared images, estimate the gaze direction and gaze point of the corresponding eyeball based on the eye features, compare the angle 'a' between the air projection position, the eye, and the horizontal line with the angle 'b' between the gaze direction and the horizontal line, and determine that the corresponding eyeball is gazing at the air projection position when the difference between angle 'a' and angle 'b' is less than the error threshold. Based on the distance from the gaze point to the corresponding eyeball, the position of the eyeball that is gazing at the air projection position is matched with the vehicle seat, and eyeball tags are assigned to all eyes gazing at the air projection position according to the corresponding seat attributes. If no air projection operation command is given, the imaging surface of the air projection will be in the preset position. If an air projection operation command is given, the angle and imaging area of ​​the air projection are adjusted based on the response priority rules, the voice request information, vehicle status information, and / or eye tag information.

2. The air projection control method based on eye tracking according to claim 1, characterized in that, Detect voice request information, including: Obtain the voice wake-up signal, and obtain the voice request information to activate the air projection function based on the voice wake-up signal; Identify the microphone array sound zone where the voice wake-up signal is located, and determine the seat of the passenger who made the voice request based on the sound zone identification result.

3. The air projection control method based on eye tracking according to claim 1, characterized in that, The vehicle status information includes at least: the vehicle's driving mode information and vehicle speed, wherein the driving mode information includes active driving mode or assisted driving mode.

4. The air projection control method based on eye tracking according to claim 1, characterized in that, The response priority rules include: Voice request priority > vehicle status priority > eye tag priority.

5. The air projection control method based on eye tracking according to claim 1, characterized in that, The preset position is where the air projection position faces the driver's seat, and the imaging area is located in the calibrated position most suitable for the driver's eye to view from the driver's seat.

6. A method for controlling air projection based on eye tracking according to any one of claims 1 to 5, characterized in that, The step of adjusting the angle and imaging area of ​​the air projection based on the response priority rule, according to the voice request information, vehicle status information, and / or eye tag information, includes: Obtain the standard straight-line distance A, standard vertical height difference B, and standard lateral distance C between the imaging area of ​​the air projection and the eyeballs at each seat; To determine the type of the current air projection operation command, consider the following principle: voice request priority > vehicle status priority > eye tag priority. If a voice request is currently in effect, the passenger seat at the location of the sound source is determined based on the voice request information, and the eye tag priority of the passenger in that seat is set to the highest. If there is no voice request, the vehicle's driving mode information and speed are obtained based on the vehicle status information. When the driving mode information is active driving mode, the driver's eye tag is masked; when the driving mode information is assisted driving mode, the driver's eye tag priority is set after the priority of the eye tags of other seats; when the vehicle speed is 0, the driver's eye tag priority is set before the priority of the eye tags of other seats. If the eyes of other seats at multiple gaze air projection positions are currently tracked, the eye tag with the highest priority is used as the target eye tag; among them, the priority order of eye tags for seats other than the driver's seat is: front passenger seat > right rear seat > left rear seat > middle rear seat; Calculate the straight-line distance A1 from the corresponding eyeball to the imaging position based on the target eyeball label, compare the straight-line distance A1 with the standard straight-line distance A, and adjust the magnification or reduction of the imaging area based on the straight-line distance comparison result; Calculate the vertical height difference B1 from the target eyeball to the imaging position based on the target eyeball label, compare the vertical height difference B1 with the standard vertical height difference B, and adjust the longitudinal tilt angle of the imaging area based on the vertical height difference comparison result. Calculate the lateral distance C1 from the target eyeball to the imaging position based on the target eyeball label, compare the lateral distance C1 with the standard lateral distance C, and adjust the lateral tilt angle of the imaging area based on the comparison result.

7. An eye-tracking-based air projection control system, characterized in that, include: The acquisition module is used to detect the presence of an air projection operation command, which includes voice request information, vehicle status information, and / or eye tag information; wherein, detecting eye tag information includes: Acquire real-time video sequences inside the vehicle, track facial information in the real-time video sequences, and identify whether eye features are present in the facial information; When eye features are identified, the human eye is illuminated with low-power active near-infrared light and a sensor array is used to capture near-infrared images. Identify eye features in near-infrared images, estimate the gaze direction and gaze point of the corresponding eyeball based on the eye features, compare the angle 'a' between the air projection position, the eye, and the horizontal line with the angle 'b' between the gaze direction and the horizontal line, and determine that the corresponding eyeball is gazing at the air projection position when the difference between angle 'a' and angle 'b' is less than the error threshold. Based on the distance from the gaze point to the corresponding eyeball, the position of the eyeball that is gazing at the air projection position is matched with the vehicle seat, and eyeball tags are assigned to all eyes gazing at the air projection position according to the corresponding seat attributes. The adjustment module is used to control the imaging surface of the air projection to a preset position if it is determined that there is no air projection operation command. It is also used to adjust the angle and imaging area of ​​the air projection based on the voice request information, vehicle status information and / or eye tag information, according to the response priority rules if an air projection operation command is determined to exist.

8. An electronic device, characterized in that, The device includes a memory and a processor, wherein the processor is used to execute computer management programs stored in the memory to implement the steps of the eye-tracking-based air projection control method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer management program, which, when executed by a processor, implements the steps of an eye-tracking-based air projection control method as described in any one of claims 1-6.

Images