Vehicle control method and system, electronic equipment, vehicle, medium and product

Abstract

The invention discloses a vehicle control method and system, electronic equipment, a vehicle, a computer readable storage medium and a computer program product. The invention discloses a vehicle control method. The method comprises the following steps: determining light field control data of a light field array according to sight watching data of a passenger and motion state data of a vehicle; and controlling the light field array according to the light field control data. According to the invention, the light field control data of the light field array is determined through the sight gazing data of the passenger and the motion state data of the vehicle, and the light field array is controlled through the light field control data, so that the light field light area of the light field array tracks the sight gazing area of the passenger. And the motion sickness problem of passengers in the unstable movement process of the vehicle is relieved or prevented.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of motion sickness mitigation, and in particular to a vehicle control method, a motion sickness prevention system, an electronic device, a vehicle, a computer readable storage medium and a computer program product. BACKGROUND

[0002] In the process of vehicle movement, when the information transmitted to the brain center by human visual sense is not coordinated or consistent with the information transmitted by the external environment, dizziness will occur. When passengers sit in a moving car, the visual sense of the vehicle movement of the vestibule is inconsistent with the light seen by the eyes, which will cause car sickness.

[0003] In related technologies, fixed position light sources or adjusting the global uniform illumination brightness of the light field are used to change the light entering the human eye, which cannot solve the problem of motion sickness caused by the conflict between vision and vestibular system signals when the vehicle accelerates, decelerates or turns. SUMMARY

[0004] The embodiments of the present application provide a vehicle control method, a control system, an electronic device, a vehicle, a computer readable storage medium and a computer program product.

[0005] In order to achieve the above-mentioned purpose, according to the first aspect of the present application, a vehicle control method is provided, comprising:

[0006] determining light field control data of a light field array according to the passenger's gaze data and the vehicle's motion state data;

[0007] controlling the light field array according to the light field control data.

[0008] In some embodiments, the light field control data includes first light field control data and second light field control data, and the determination of the light field control data of the light field array according to the passenger's gaze data and the vehicle's motion state data comprises:

[0009] determining the first light field control data according to the passenger's gaze data; and / or,

[0010] determining the second light field control data according to the passenger's gaze data and the vehicle's motion state data.

[0011] In some embodiments, the gaze data includes visual focus data, and the determination of the first light field control data according to the passenger's gaze data comprises:

[0012] determining the first light field control data according to the visual focus data.

[0013] In some embodiments, the light field array comprises a first light field region, and the controlling the light field array according to the light field control data comprises:

[0014] controlling the first light field region according to the first light field control data.

[0015] In some embodiments, the eye gaze data comprises visual residual light data, and the determining the second light field control data according to the eye gaze data of the passenger and the motion state data of the vehicle comprises:

[0016] determining the second light field control data according to the visual residual light data and the motion state data.

[0017] In some embodiments, the light field array comprises a second light field region, and the second light field control data comprises second light field region data and optical performance parameters, and the determining the second light field control data according to the visual residual light data and the motion state data comprises:

[0018] determining the second light field region data according to the visual residual light data;

[0019] determining the optical performance parameters according to the motion state data.

[0020] In some embodiments, the light field array comprises a second light field region, and the controlling the light field array according to the light field control data comprises:

[0021] controlling the second light field region according to the second light field region data;

[0022] controlling optical performance of the second light field region according to the optical performance parameters.

[0023] In some embodiments, the motion state data comprises at least one of a current motion direction and a current motion speed of the vehicle; and / or,

[0024] the optical performance parameters comprise at least one of a light wave propagation direction, a light wave propagation speed, and a light wave wavelength.

[0025] In some embodiments, the optical performance of the second light field region comprises at least one of a light wave propagation direction, a light wave propagation speed, and a light wave wavelength of the second light field region.

[0026] In some embodiments, before the determining the optical performance parameters of the light field array according to the eye gaze data of the passenger and the motion state data of the vehicle, the method comprises:

[0027] determining the eye gaze data according to the eye image of the passenger and the first distance;

[0028] wherein the first distance is a distance from the passenger's eye to a target plane of the vehicle.

[0029] In some embodiments, the determining the gaze fixation data according to the eye image of the passenger and the first distance comprises:

[0030] calculating a head degree-of-freedom pose of the passenger according to the eye image of the passenger;

[0031] compensating the head degree-of-freedom pose of the passenger according to the first distance to obtain the gaze fixation data.

[0032] In some embodiments, the eye image comprises a pupil contour and an eye cornea, and the calculating the head degree-of-freedom pose of the passenger according to the eye image of the passenger comprises:

[0033] calculating a line-of-sight vector of the passenger's eye according to the pupil contour and the eye cornea, and generating the head degree-of-freedom pose of the passenger according to the line-of-sight vector, wherein the degree-of-freedom pose comprises a position parameter and / or an orientation parameter in a three-dimensional space, wherein the position parameter comprises coordinate data in a three-dimensional coordinate system, and the orientation parameter comprises at least one of a pitch angle, a yaw angle, and a roll angle.

[0034] In some embodiments, the calculating the line-of-sight vector of the passenger's eye according to the pupil contour and the eye cornea comprises:

[0035] calculating the line-of-sight vector of the passenger's eye according to a pupil parameter of the pupil contour and a relative position of a corneal reflection point of the eye cornea by a convolutional neural network model.

[0036] wherein the pupil parameter of the pupil contour comprises a ratio of major and minor axes of a pupil ellipse, a tilt angle, and a center coordinate.

[0037] In some embodiments, the method further comprises, before the determining the gaze fixation data according to the eye image of the passenger and the first distance:

[0038] performing distortion correction and coordinate system alignment on the eye image in a camera coordinate system.

[0039] In some embodiments, the method further comprises, before the determining the light field control data of the light field array according to the gaze fixation data of the passenger and the motion state data of the vehicle:

[0040] processing the collected initial motion state data of the vehicle by a preset filtering algorithm to obtain the motion state data of the vehicle.

[0041] In some embodiments, the method further comprises:

[0042] processing the initial motion state data of the vehicle by a preset timestamp alignment rule to eliminate time deviation of each data in the initial motion state data.

[0043] In some embodiments, the initial motion state data comprises at least one of yaw rate, speed, acceleration and steering wheel angle.

[0044] According to a second aspect of the present application, a vehicle control system is provided, comprising:

[0045] a controller and a light field array, the controller and the light field array being connected;

[0046] the controller is configured to determine light field control data of the light field array according to gaze data of a passenger and motion state data of a vehicle;

[0047] and control the light field array according to the light field control data.

[0048] In some embodiments, the controller comprises:

[0049] a first data processing module configured to determine the gaze data of the passenger according to an eye image of the passenger and a first distance;

[0050] wherein the first distance is a distance from the eye of the passenger to a target plane of the vehicle.

[0051] In some embodiments, the controller further comprises:

[0052] an eye tracking module connected to the first data processing module and configured to obtain the eye image of the passenger and the first distance.

[0053] In some embodiments, the eye tracking module comprises:

[0054] a plurality of cameras configured to obtain the eye image of the passenger;

[0055] a distance sensor configured to obtain the first distance.

[0056] In some embodiments, the controller comprises:

[0057] a second data processing module configured to determine the motion state data of the vehicle.

[0058] In some embodiments, the light field array comprises:

[0059] An LED array comprising a microstructured diffusion film.

[0060] According to a third aspect of the present application, an electronic device is provided, comprising:

[0061] a memory and a processor;

[0062] The memory stores a computer program, and the processor is configured to execute the computer program stored in the memory to perform the vehicle control method according to the first aspect of the present application.

[0063] According to a fourth aspect of the present application, a vehicle is provided, comprising the electronic device according to the third aspect of the present application, or comprising the vehicle control system according to the second aspect of the present application.

[0064] According to a fifth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is configured to be executed by a processor to perform the vehicle control method according to the first aspect of the present application.

[0065] According to a sixth aspect of the present application, a computer program product is provided, which comprises a computer program or instructions, and the computer program or instructions are configured to be executed by a processor to implement the vehicle control method according to the first aspect of the present application.

[0066] The present application determines the light field control data of the light field array by the gaze data of the passenger and the motion state data of the vehicle, and controls the light field array by the light field control data, so that the light field bright light area of the light field array tracks the gaze area of the passenger, and the optical performance parameters of the light field bright light area are adjusted according to the change of the motion state data, effectively alleviating or preventing the motion sickness problem of the passenger during the unstable motion of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0067] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0068] Figure 1 A flowchart of a vehicle control method according to an embodiment of the present application;

[0069] Figure 2 An architecture diagram of a vehicle control system according to an embodiment of the present application;

[0070] Figure 3 An architecture diagram of another vehicle control system according to an embodiment of the present application;

[0071] Figure 4A structural schematic diagram of an eye tracking device according to an embodiment of the present application;

[0072] Figure 5 A structural schematic diagram of a light field array according to an embodiment of the present application;

[0073] Figure 6 A structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the description and the claims of the present application and the above description of the drawings, the terms "comprising" and "having" and any variations thereof, are intended to cover not exclusively containing.

[0076] In the description of the embodiments of the present application, the technical terms "first", "second", and the like are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0077] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0078] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects have an "or" relationship.

[0079] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0080] In the description of the embodiments of this application, unless otherwise expressly specified and limited, the technical terms such as "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or a connection through a network. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application according to the specific circumstances.

[0081] Dizziness occurs when the information transmitted from the human visual system to the brain is inconsistent with the information transmitted from the external environment during the movement of a vehicle. Specifically, motion sickness occurs when the vestibular system perceives a discrepancy between the light emitted by the vehicle and the light seen by the eyes while riding in a moving vehicle.

[0082] To address the aforementioned problems, this application provides a vehicle control method, such as... Figure 1 As shown, it includes:

[0083] S101: Determine the light field control data of the light field array based on passenger gaze data and vehicle motion state data;

[0084] S102: Control the light field array according to the light field control data.

[0085] During the movement of a vehicle, the acceleration, deceleration, turning, or swaying or rocking of the vehicle when passing over unpaved roads can cause the passenger's body to dynamically shift or move along with the vehicle. The passenger's gaze will also change with the dynamic shift or movement of the body, resulting in frequent changes in the light received by the passenger's eyes. When a passenger is riding in a moving vehicle, the discrepancy between the light perceived by the visual vestibule of the moving vehicle and the light seen by the eyes can cause motion sickness.

[0086] In order to keep the light sensation of the passenger's visual vestibular sensation consistent with the light seen by the eyes during the movement of the vehicle, the application sets a light field array in the passenger's eye view range. Generally, the light field array is installed in the passenger's eye view range, such as setting the light field array in the front of the passenger's field of view, in the top lining of the vehicle, covering an area of about 0.8 square meters. The light field array can adjust the projection angle of the light field array according to the passenger's sitting posture, ensuring that the light coverage range matches the passenger's field of view. By dynamically controlling the light field array to track the passenger's field of view, the light in the passenger's field of view is compensated, so that when the light projected by the light field array enters the passenger's field of view during the movement of the vehicle, the light entering the passenger's eyes is consistent with the light sensation of the passenger's visual vestibular sensation, thereby effectively alleviating or preventing the motion sickness problem of the passenger during the unstable movement of the vehicle.

[0087] In order to effectively control the light of the light field array to follow the position of the passenger's eyes and the position of the eye light in the movement of the vehicle in real time, the passenger's gaze data and the motion state data of the vehicle in the movement process need to be obtained. The motion state data will change according to the real-time movement of the vehicle, and accordingly, the passenger's gaze data will also change according to the motion state data. By obtaining the passenger's gaze data and the motion state data of the vehicle in the movement process, and determining the light field control data of the light field array according to the gaze data and the motion state data, and then controlling the light field array according to the light field control data, the light field bright light area of the light field array and the optical performance of the light field bright light area are highly matched with the passenger's gaze area during the actual movement of the vehicle, thereby alleviating the conflict between the passenger's vision and the vestibular system.

[0088] The application determines the light field control data of the light field array through the passenger's gaze data and the motion state data of the vehicle, and controls the light field array through the light field control data, so that the light field bright light area of the light field array tracks the passenger's gaze area, and the optical performance parameters of the light field bright light area are adjusted according to the change of the motion state data, effectively alleviating or preventing the motion sickness problem of the passenger during the unstable movement of the vehicle.

[0089] The light field array is usually composed of a micro light emitting diode (Mini-LED) array. Specifically, the light field array is composed of a plurality of independently controllable Mini-LED units arranged in an M-row-by-N-column arrangement to form a rectangular array, a diamond array, a triangular array, an array arranged in a circular pattern from the inside to the outside, an S-shaped array, or the like. The specific arrangement manner is not limited again. The light field array further includes a driving board integrated with a plurality of pulse width modulation (PWM) controllers for controlling the plurality of independently controllable Mini-LED units. Each PWM controller can support 16-bit precision brightness adjustment, and the response speed can reach the microsecond level.

[0090] In some embodiments, the light field control data includes first light field control data and second light field control data, and the light field control data of the light field array is determined according to the gaze data of the passenger and the motion state data of the vehicle, including:

[0091] The first light field control data is determined according to the gaze data of the passenger; and / or,

[0092] The second light field control data is determined according to the gaze data of the passenger and the motion state data of the vehicle.

[0093] Optionally, the light field control data includes first light field control data and second light field control data. When determining the light field control data of the light field array according to the gaze data of the passenger and the motion state data of the vehicle, the first light field control data can be determined by the gaze data of the passenger, and / or the second light field control data can be determined according to the gaze data of the passenger and the motion state data of the vehicle. The first light field control data and the second light field control data are both control data for controlling the light field array, and the light field array can be controlled by the light field control data at the same time.

[0094] In some embodiments, the gaze data includes visual focus data, and the first light field control data is determined according to the gaze data of the passenger, including:

[0095] The first light field control data is determined according to the visual focus data.

[0096] Optionally, the gaze fixation data includes visual focus data. The visual focus data is focus data of the passenger's eye pupil fixation, used to determine a visual focus area of the passenger's eye, the visual focus area being an area where the passenger's eye pupil focus is fixed, such as the landing position of the passenger's pupil focus and / or the range of the fixation area, etc. The visual focus data can be marked by two-dimensional coordinates. The first light field control data can be determined according to the visual focus data, and the first light field control data generally includes a light bright area range formed by the number of Mini-LED units in the light field array that matches the visual focus area; the first light field control data also includes control parameters for controlling the light bright area range, such as the brightness, projection angle, etc. of the Mini-LED unit.

[0097] In some embodiments, the light field array includes a first light field area, and the controlling the light field array according to the light field control data includes:

[0098] controlling the first light field area according to the first light field control data.

[0099] Optionally, the light field array includes a first light field area, and the controlling the light field array according to the light field control data includes: controlling the first light field area by the first light field control data. The PWM controller can directly determine the first light field area of the light field array by the first light field control data, and the first light field area is a light bright area range formed by the number of Mini-LED units in the first light field control data that matches the visual focus area. The PWM controller controls to turn on all Mini-LED units in the first light field area, and adjusts the brightness of the Mini-LED units according to the brightness parameter in the first light field control data, and adjusts the angle of the Mini-LED units according to the projection angle control in the first light field control data.

[0100] The brightness parameter of the Mini-LED unit adjusts the brightness of the Mini-LED unit, and the projection angle control adjusts the angle of the Mini-LED unit. The Mini-LED units not in the first light field area are in a dormant state.

[0101] In some embodiments, the gaze fixation data includes visual residual light data, and the determining the second light field control data according to the gaze fixation data of the passenger and the motion state data of the vehicle includes:

[0102] The second light field control data is determined according to the visual residual light data and the motion state data.

[0103] Optionally, the gaze fixation data also includes visual residual light data. The visual residual light data includes data when the passenger's gaze is uncertainly shifted or the residual light is swept due to the passenger's body or head shaking or dynamic offset caused by the change of the motion state data of the vehicle. The visual residual light data includes a visual residual light area, which is an area where the passenger's visual residual light is swept during acceleration, deceleration, shaking, and turn signal process of the vehicle.

[0104] In order to enable the light waves of the light field array to track the visual periphery of the human eye and compensate for the light rays of the visual periphery of the human eye, second light field control data needs to be determined based on the visual periphery data and the motion state data of the vehicle. The second light field control data includes coordinate range data of the passenger visual periphery in a two-dimensional coordinate system, and speed, direction, position, and range data of changes in the passenger visual periphery range during acceleration, deceleration, shaking, and turning light of the vehicle.

[0105] In some embodiments, the second light field control data includes second light field region data and optical performance parameters, and the determining the second light field control data based on the visual periphery data and the motion state data includes:

[0106] determining the second light field region data based on the visual periphery data;

[0107] determining the optical performance parameters based on the motion state data.

[0108] Optionally, the second light field control data includes second light field region data and optical performance parameters. The second light field region data can be determined based on the visual periphery data, and the second light field region data generally includes a light bright region range formed by the number of Mini-LED units in the light field array that match the visual periphery region.

[0109] Meanwhile, the optical performance parameters can be determined based on the motion state data of the vehicle. The optical performance parameters are parameters for controlling the optical performance of the Mini-LED units in the region range in the light field array that matches the second light field region data.

[0110] Optionally, the motion state data includes at least one of a current motion direction and a current motion speed of the vehicle; and / or the optical performance parameters include at least one of a light wave propagation direction, a light wave propagation speed, and a light wave wavelength.

[0111] The motion state data includes at least one of a current motion direction and a current motion speed of the vehicle. The current motion direction and the current motion speed of the vehicle are usually absolute motion direction and absolute motion speed of the vehicle during acceleration, deceleration, turning, or shaking on non-paved road, and are calculated based on key parameters such as speed, current angular speed, longitudinal / lateral acceleration, steering wheel angle, and the like during motion of the vehicle. During shaking of the vehicle, the head of the passenger also moves with the shaking of the vehicle. The current motion direction and the current motion speed of the vehicle can be reflected in the motion direction and the motion speed of the glint of the passenger when the glint dynamically shifts with the motion direction and the motion speed of the vehicle. Therefore, the optical performance parameter includes at least one of a light wave propagation direction, a light wave propagation speed, and a light wave wavelength of a light field array region in the light field array that is adapted to the visual glint data.

[0112] In general, the current motion direction of the vehicle is opposite to the direction in which the glint of the passenger shifts, and therefore, the light wave propagation direction is opposite to the current motion direction of the vehicle, the light wave propagation speed is consistent with the current motion speed of the vehicle, and the wavelength of the light wave is dynamically adjusted according to the current motion speed.

[0113] In some embodiments, the light field array includes a second light field region, and the controlling the light field array according to the light field control data includes:

[0114] controlling the second light field region according to the second light field region data;

[0115] controlling an optical performance of the second light field region according to the optical performance parameter.

[0116] Optionally, the light field array includes a second light field region. The controlling the light field array according to the light field control data further includes: controlling the second light field region according to second light field region data. The PWM controller can directly determine the second light field region of the light field array through the second light field region data. The second light field region is a light range formed by the number of Mini-LED units in the second light field region data that matches the visual glint region. The visual glint of the passenger has obvious directionality, and the visual glint region range is usually large. However, the area of the light field array can only match the internal space of the vehicle, and is usually small. Therefore, the range of the second light field region is usually a local region in the light field array. For example, the visual glint of the passenger is in the left direction of the light field array, and the second light field region is the entire region or a local region to the left of the first light field array in the light field array. The range of the second light field region is determined according to the area of the light field array and the amplitude of the visual glint.

[0117] Meanwhile, the optical performance of the second light field region is controlled according to the optical performance parameters. In the second light field region, the PWM controller controls the optical performance of the second light field region according to the data of the light wave propagation direction, the light wave propagation speed, and the light wave wavelength of the optical performance parameters.

[0118] Optionally, the optical performance of the second light field region includes at least one of the light wave propagation direction, the light wave propagation speed, and the light wave wavelength of the second light field region.

[0119] Specifically, the PWM controller controls the light ray of the Mini-LED unit in the second light field region to change regularly according to the optical performance parameters, such as controlling the light wave propagation direction of the Mini-LED unit in the second light field region to follow the movement direction of the visual afterimage, the light wave propagation speed to match the movement speed of the visual afterimage, and the light wave wavelength to adjust with the change of the movement speed of the visual afterimage. The light wave propagation direction, the light wave propagation speed, and the light wave wavelength of the dynamic light wave formed by the Mini-LED unit in the second light field region can be dynamically adjusted according to the optical performance parameters in real time, matched with the passenger's body acceleration, to dynamically compensate the visual afterimage of the passenger and relieve or prevent the passenger from dizziness.

[0120] According to the above-mentioned embodiments, in the process of controlling the light field array according to the light field control data, the dynamic light field reconstruction of the light field array is essentially performed. The light field array of the present application adopts a double-buffer rendering mechanism to realize dynamic light field reconstruction. The dynamic light field reconstruction refers to real-time calculation and refreshing of the first light field region and the second light field region to be turned on in the light field array, and real-time adjustment of the light intensity distribution and the movement form in the light wave propagation process of the Mini-LED unit in the first light field region and / or the second light field region, according to the visual focus data and / or the visual afterimage data of the passenger and the motion state data of the vehicle during the operation of the vehicle. Preferably, the global light field update of the light field array and / or the update of the light intensity distribution and the movement form in the light wave propagation process of the first light field region and the second light field region are completed once every 10 ms.

[0121] Optionally, the background layer of the light field array dynamically adjusts the global brightness of the light field array according to the ambient light sensor data: in the daytime mode, the global brightness of the light field array is maintained at 500 nits of basic brightness to offset the interference of external strong light on the eyes of the passenger; in the night mode, the global brightness of the light field array is maintained at 80 nits of basic brightness to avoid dazzling the eyes of the passenger. The foreground layer applies a gradient mask with a radius of 15 cm in the first light field region, the central brightness of the first light field region is attenuated to 30%, and the edge of the first light field region is smoothly transitioned to 100% through a Gaussian blur algorithm, to ensure that there is no abrupt brightness change in the visual focus area of the passenger; by attenuating the central brightness of the first light field region to 30%, the interference of the central brightness of the first light field region on the visual focus of the passenger's eyes can be obviously avoided.

[0122] For the visual peripheral light area of the passenger, the PWM controller controls the light wave propagation in the second light field area in the direction opposite to the motion direction of the vehicle, the light wave form simulates the fluid motion characteristics through the Navier-Stokes equation, the light wave propagation speed is linearly related to the speed of the vehicle (the proportional coefficient is 0.8 m / s per ° / s), and the wavelength of the light wave is dynamically adjusted according to the speed of the vehicle, so that the light wave propagation motion track in the second light field area matches the acceleration of the peripheral light transfer of the passenger. Optionally, the reference wavelength of the light wave of the Mini-LED unit is 10 cm, and in the dynamic reconstruction process, the speed of the vehicle increases by 10 km / h, and the light wave of the Mini-LED unit shortens by 2 cm. This real-time synchronization of the light wave parameters of the second light field area and the motion state data of the vehicle can improve the compensation accuracy of the light field array for the visual peripheral light area of the passenger's eye, and the motion sickness relief efficiency is improved by 67%.

[0123] According to the experimental data, in the dynamic light field reconstruction process, only 15% of the Mini-LED units in the light field array need to be controlled to be always on, which can reduce the overall energy consumption of the light field array by 85%.

[0124] Optionally, the controller writes the light field control data into the display memory in real time, and directly transmits the light field control data to the Light-Emitting Diode (LED) drive board of the light field array through a Direct Memory Access (DMA) channel, avoiding the delay of data transmission caused by the intervention of the Central Processing Unit (CPU) of the controller.

[0125] In some embodiments, before determining the optical performance parameters of the light field array according to the gaze data of the passenger and the motion state data of the vehicle, the method comprises:

[0126] determining the gaze data according to the eye image of the passenger and the first distance;

[0127] wherein the first distance is the distance from the passenger's eye to the target plane of the vehicle.

[0128] Before determining the optical performance parameters of the light field array according to the gaze data of the passenger and the motion state data of the vehicle, the eye image of the passenger and the first distance need to be obtained, and the gaze data is determined according to the obtained eye image of the passenger and the first distance. The first distance is the distance from the passenger's eye to the target plane of the vehicle, and the target plane of the vehicle is usually the area that the passenger's field of view often gazes at, such as the top or front inner wall of the vehicle.

[0129] Optionally, the application obtains the eye image and the first distance of the passenger by an eye tracking device. Specifically,

[0130] The eye tracking device is composed of multiple groups of shooting heads and distance sensors. The eye image of the passenger is obtained by the multiple groups of shooting heads, and the distance from the eye of the passenger to the target plane of the vehicle is measured by the distance sensors.

[0131] In some embodiments, before determining the gaze fixation data according to the eye image and the first distance of the passenger, the method comprises:

[0132] The eye image is corrected for distortion and aligned with the coordinate system in the camera coordinate system.

[0133] Optionally, before determining the gaze fixation data according to the eye image and the first distance of the passenger, the eye image needs to be corrected for distortion and aligned with the coordinate system in the camera coordinate system. Multiple cameras are arranged at different positions, and the same scene point observed by multiple cameras has an independent image coordinate system for each camera. After the multiple cameras each obtain the eye image of the passenger, the eye image of the passenger obtained by each camera needs to be mapped to the same unified coordinate system, and the eye image needs to be corrected for distortion and aligned with the coordinate system, so as to eliminate the eye image data deviation caused by light, position, shooting angle, etc. of the eye image obtained by different cameras.

[0134] In some embodiments, determining the gaze fixation data according to the eye image and the first distance of the passenger comprises:

[0135] Calculating the head degree of freedom pose of the passenger according to the eye image of the passenger;

[0136] Compensating the head degree of freedom pose of the passenger according to the first distance to obtain the gaze fixation data.

[0137] Optionally, determining the gaze fixation data according to the eye image and the first distance of the passenger first needs to calculate the head degree of freedom pose of the passenger according to the eye image of the passenger, and then compensate the head degree of freedom pose of the passenger according to the first distance to obtain the gaze fixation data.

[0138] In some embodiments, the eye image comprises the pupil contour and the cornea of the eye, and calculating the head degree of freedom pose of the passenger according to the eye image of the passenger comprises:

[0139] According to the pupil contour and the corneal reflection point of the eye, a line-of-sight vector of the passenger's eye is calculated, and a head degree-of-freedom pose of the passenger is generated according to the line-of-sight vector, wherein the degree-of-freedom pose includes a position parameter and / or an orientation parameter in a three-dimensional space, and the position parameter includes coordinate data in a three-dimensional coordinate system, and the orientation parameter includes at least one of a pitch angle, a yaw angle and a roll angle.

[0140] Optionally, the eye image includes a pupil contour and a corneal reflection point of the eye, and a line-of-sight vector of the passenger's eye is usually calculated through the pupil contour and the corneal reflection point of the eye, and a head degree-of-freedom pose of the passenger is generated according to the line-of-sight vector. The line-of-sight vector of the passenger's eye is a three-dimensional vector in a three-dimensional coordinate system, for example, coordinate data in an X-axis, a Y-axis and a Z-axis in a three-dimensional coordinate system, which is gazed by the passenger's eyeball. The head degree-of-freedom pose of the passenger includes a position parameter and / or an orientation parameter in a three-dimensional space. The position parameter includes coordinate data of the passenger's head in a three-dimensional coordinate system, and the coordinate data is coordinate data in an X-axis, a Y-axis and a Z-axis in a three-dimensional coordinate system. The orientation parameter includes at least one of a pitch angle, a yaw angle and a roll angle of the passenger's head (especially the eye) in a three-dimensional coordinate system. The pitch angle is usually an angle in an X-axis and a Y-axis plane, the yaw angle is usually an angle in an X-axis and a Z-axis plane, and the roll angle is usually an angle in a Y-axis and a Z-axis plane.

[0141] In some embodiments, the line-of-sight vector of the passenger's eye is calculated according to the pupil contour and the corneal reflection point of the eye, including:

[0142] The line-of-sight vector of the passenger's eye is calculated according to the pupil parameters of the pupil contour and the relative positions of the corneal reflection points of the cornea of the eye through a convolutional neural network model.

[0143] The pupil parameters of the pupil contour include a long-short axis ratio, an inclination angle and a center coordinate of a pupil ellipse.

[0144] Optionally, the line-of-sight vector of the passenger's eye is calculated according to the pupil parameters of the pupil contour and the relative positions of the corneal reflection points of the cornea of the eye through a convolutional neural network model, and the pupil parameters of the pupil contour include a long-short axis ratio, an inclination angle and a center coordinate of a pupil ellipse. The convolutional neural network model includes a lightweight convolutional neural network, and an improved YOLOv5s model is used to calculate the line-of-sight vector of the passenger's eye.

[0145] In the eyeball tracking device, three groups of 850nm near-infrared cameras and Time of Flight (ToF) sensors are used to form a triangular array, each camera is built-in with a global shutter sensor, and the eye image of the passenger is captured at a rate of 120 frames / second, and a high-contrast image of the pupil contour and the corneal reflection point of the cornea of the eye in the eye image is obtained.

[0146] Then, the pupil profile collected by the three-camera head is corrected for distortion and aligned with the coordinate system of the high-contrast image of the corneal reflection point of the eye cornea, and the line-of-sight vector is calculated by using the long and short axis ratio, tilt angle and center coordinates of the pupil ellipse, combined with the relative position of the corneal reflection point.

[0147] And through the three-camera parallax principle, the line-of-sight vector of the passenger's eye is calculated in real time, the line-of-sight vector is converted into a three-dimensional gaze vector, and the pupil ellipse parameters in the high-contrast image of the pupil profile and the corneal reflection point of the eye cornea are detected by using the improved YOLOv5s model, and the head degree of freedom pose of the passenger is calculated according to the three-dimensional gaze vector by using the Perspective-n-Point (PnP) algorithm.

[0148] The PnP algorithm maps the features of the high-contrast image of the pupil profile and the corneal reflection point of the eye cornea (two-dimensional image) to the three-dimensional space according to the pre-calibrated camera internal and external parameter matrices, calculates the head degree of freedom pose of the passenger, and compensates for the visual gaze point offset caused by head shaking. Finally, the two-dimensional coordinate data of the visual gaze point is output, and the two-dimensional coordinate data of the visual gaze point is fused with the historical visual gaze point data obtained last time by using the Bayesian filtering algorithm, so as to improve the positioning stability of the two-dimensional coordinate data of the visual gaze point and avoid the jump of the two-dimensional coordinate data of the visual gaze point caused by blinking or temporary line-of-sight obstruction. At the same time, the ToF sensor measures the first distance from the passenger's eye to the target plane of the vehicle at a frequency of 60Hz, compensates for the mapping deviation of the two-dimensional coordinate data of the visual gaze point caused by the forward and backward displacement of the passenger's head, and finally outputs the visual focus data in the two-dimensional coordinate system. This way makes the two-dimensional coordinate accuracy of the visual focus data reach 0.3° angle of view error, which meets the sub-degree level eye tracking requirement.

[0149] In some embodiments, before determining the light field control data of the light field array according to the passenger's gaze fixation data and the motion state data of the vehicle, the method further comprises:

[0150] The initial motion state data of the vehicle is processed by a preset dynamic model to obtain the motion state data of the vehicle.

[0151] Optionally, before determining the light field control data of the light field array according to the passenger's gaze fixation data and the motion state data of the vehicle, the initial motion state data of the vehicle is obtained, and the initial motion state data of the vehicle is processed by a preset dynamic model to obtain the motion state data of the vehicle.

[0152] In some embodiments, the initial motion state data includes at least one of yaw rate, speed, acceleration, and steering wheel angle.

[0153] Optionally, the initial motion state data is collected as motion data of the previous 0.5 seconds at the current time. The data interface of the vehicle collects the initial motion state data of the vehicle of the previous 0.5 seconds at the current time in real time through the sensors at a data collection frequency of 100 Hz. The initial motion state data includes at least one of yaw rate, speed, acceleration, and steering wheel angle.

[0154] In some embodiments, the method further includes:

[0155] The initial motion state data of the vehicle is processed by a preset timestamp alignment rule to eliminate the time deviation of each data in the initial motion state data.

[0156] Optionally, during the collection of the initial motion state data of the vehicle through the sensors, noise data exists in the initial motion state data due to the differences in the data collection methods and the time differences of the data collection between different sensors, which affects the consistency of the initial motion state data.

[0157] Therefore, the application adopts a sliding window weighted average filtering algorithm to smooth the original signals collected by the sensors, and eliminates the time deviation of the multi-source initial motion state data obtained by different sensors through a timestamp alignment mechanism. For example, the steering wheel angle data and the yaw rate in the initial motion state data are processed by a preset timestamp alignment rule to control the time error of the steering wheel angle data and the yaw rate within ±5 ms, and the phase difference between the steering wheel angle data and the yaw rate ensures the spatiotemporal consistency of the initial motion state data.

[0158] Then, the initial motion state data of the vehicle is input into a second-order dynamics model. The second-order dynamics model of the vehicle is based on a second-order state space model of the dynamics of the vehicle, and the historical initial motion state data of the previous 0.5 seconds is input to predict the motion trend of the vehicle within 100 milliseconds in the future through a state space equation. For example, when the steering wheel angle rate of the vehicle is detected to be greater than 2° / s, the second-order dynamics model calculates the expected yaw rate increment according to the wheelbase and the height of the center of mass of the vehicle, and dynamically adjusts the predicted motion state data of the vehicle, such as the current motion direction and / or the current motion speed of the vehicle, in combination with the road adhesion coefficient. The prediction result is converted into an optical performance parameter of a second light field region in the light field array, wherein the optical performance parameter includes data such as light wave propagation direction, light wave propagation speed, and wavelength.

[0159] The application also provides a vehicle control system, which comprises:

[0160] a controller and a light field array, the controller and the light field array being connected;

[0161] the controller is configured to determine light field control data of the light field array according to eye gaze data of a passenger and motion state data of a vehicle;

[0162] and control the light field array according to the light field control data.

[0163] As shown in Figure 2 the vehicle control system includes a controller 100 and a light field array 200, the controller 100 and the light field array 200 are connected, and the controller is configured to determine light field control data of the light field array according to eye gaze data of a passenger and motion state data of a vehicle.

[0164] In some embodiments, the light field array 200 includes: an LED array, the LED array includes a microstructure diffusion film.

[0165] The light field array 200 is generally composed of an LED array, specifically, the light field array 200 is composed of a plurality of independently controllable Mini-LED units, which are arranged in an M-row-by-N-column arrangement to form a rectangular array, a diamond array, a triangular array, an array arranged in a circular pattern from the inside to the outside, an S-shaped array, etc. The specific arrangement is not limited again. As shown in Figure 5 In one example of the present application, the light field array 200 is composed of 1024 independently controllable Mini-LED units, each unit has a size of 2mm x 2mm, a center-to-center distance of 4mm, and a peak brightness of the light field array reaches 500 nits using flip-chip packaging technology.

[0166] The LED array surface is also covered with a microstructure diffusion film. The microstructure diffusion film forms a gradient refractive index layer through a precision etching process, which diffuses the point light source in the light field array 200 into a uniform area light source, while retaining

[0167] the single-pixel light control capability of the Mini-LED array. At the same time, the refractive index gradient design of the microstructure diffusion film refers to the cell distribution density of the fovea centralis (high sensitivity area) and peripheral area (low sensitivity area) of the human retina, so that the light field brightness decay curve matches the human retina photoreceptor response characteristics.

[0168] The light field array 200 also includes a driving board, which integrates 1024 PWM controllers to control the 1024 independently controllable Mini-LED units. Each PWM controller can support 16-bit precision brightness adjustment, and the response speed can reach 10 microseconds.

[0169] The light field array 100 is connected with the controller 100 through a serial peripheral interface (SPI) bus, and receives light field control data sent from the controller 100 through the SPI bus.

[0170] The light field array 200 of the present application can quickly control 1024 independently controllable Mini-LED units in the light field array according to the light field control data to form different light areas for light interference printing and dynamic light wave compensation of the passenger's visual line. The microsecond-level response speed can quickly respond to the Mini-LED unit control instructions, and the light area in the light field array 200 can be quickly adjusted according to the motion state of the vehicle and the human eye fixation area to meet the trend of high matching between the light field array 200 and the motion trend of the vehicle and the trend of the human eye fixation area changing or changing, thereby relieving the perception conflict between vision and vestibular system.

[0171] In some embodiments, the controller comprises:

[0172] a first data processing module configured to determine gaze fixation data of the passenger according to the eye image of the passenger and a first distance;

[0173] The first distance is the distance from the passenger's eye to the target plane of the vehicle.

[0174] Optionally, as shown in Figure 3 The controller 100 comprises a first data processing module 101 configured to determine gaze fixation data of the passenger according to the eye image of the passenger and a first distance, wherein the first distance is the distance from the passenger's eye to the target plane of the vehicle.

[0175] In some embodiments, the system further comprises:

[0176] an eye tracking device connected with the first data processing module and configured to acquire the eye image of the passenger and the first distance.

[0177] Optionally, the vehicle control system further comprises an eye tracking device 300 connected with the first data processing module 101, the eye tracking device being configured to acquire the eye image of the passenger and the first distance and send the eye image of the passenger and the first distance to the first data processing module 101 for data processing.

[0178] In some embodiments, the eye tracking device comprises:

[0179] Multiple cameras, wherein the multiple cameras are used to acquire images of the passenger's eyes;

[0180] A distance sensor, used to acquire the first distance.

[0181] Optionally, the eye-tracking device 300 includes multiple cameras 10 and a distance sensor 20. The multiple cameras 10 are used to acquire images of the passenger's eyes, and the distance sensor 20 is used to acquire a first distance.

[0182] like Figure 5 As shown, the eye-tracking device 300 of this application employs a triangular array composed of three sets of 850nm near-infrared cameras and a ToF sensor. Each camera has a built-in global shutter sensor and captures images of the passenger's eyes at a rate of 120 frames per second, obtaining high-contrast images of the pupil outline and corneal reflection points in the eye images. Simultaneously, the ToF sensor measures the first distance from the passenger's eyes to the target plane of the vehicle at a frequency of 60Hz.

[0183] The eye-tracking device 300 is connected to the first data processing module 101 via a USB 3.0 interface, and transmits the eye image and the first distance to the first data processing module 101 via the USB 3.0 interface. The first data processing module 101 has a built-in field-programmable gate array (FPGA). The first data processing module 101 performs hardware-level data preprocessing in the FPGA, including eye image denoising, feature point extraction and coordinate normalization.

[0184] Specifically, in the first data processing module 101, the high-contrast image of the pupil outline captured by the camera and the corneal reflection point of the eye's cornea is subjected to distortion correction and coordinate system alignment. The line of sight vector is calculated by using the ratio of the major and minor axes, the tilt angle and the center coordinates of the detected pupil ellipse, combined with the relative position of the corneal reflection point.

[0185] By utilizing the trinocular parallax principle, the system calculates the passenger's gaze vector in real time, transforms the gaze vector into a three-dimensional gaze vector, and combines it with an improved YOLOv5s model to detect the pupil ellipse parameters in high-contrast images of the pupil contour and corneal reflection points. Simultaneously, the PnP algorithm is used to calculate the passenger's head pose based on the three-dimensional gaze vector.

[0186] The PnP algorithm maps the high-contrast image (two-dimensional image) features of the pupil contour and the corneal reflection point of the eye cornea to the three-dimensional space according to the pre-calibrated camera internal and external parameter matrices, calculates the head degree of freedom pose of the passenger, and compensates for the visual gaze point offset caused by head shaking. The final output is the two-dimensional coordinate data of the visual gaze point, and the two-dimensional coordinate data of the visual gaze point is fused with the historical two-dimensional coordinate data of the visual gaze point obtained last time through the Bayesian filtering algorithm, so as to improve the positioning stability of the two-dimensional coordinate data of the visual gaze point and avoid the jump of the two-dimensional coordinate data of the visual gaze point caused by blinking or temporary line of sight obstruction. At the same time, the first distance compensates for the mapping deviation of the two-dimensional coordinate data of the visual gaze point caused by the forward and backward displacement of the passenger's head, and finally outputs the visual focus data in the two-dimensional coordinate system. This way makes the two-dimensional coordinate accuracy of the visual focus data reach 0.3° angle of view error, which meets the sub-degree level eye tracking demand.

[0187] In some embodiments, the controller comprises:

[0188] The second data processing module is configured to acquire initial motion state data of the vehicle transmitted by the sensor, and determine motion state data of the vehicle according to the initial motion state data.

[0189] Optionally, the controller 100 further comprises a second data processing module 102, which is configured to acquire initial motion state data of the vehicle transmitted by the sensor 400, and determine motion state data of the vehicle according to the initial motion state data. The sensor 400 of the vehicle is connected to the second data processing module 102, the sensor 400 collects the initial motion state data of the vehicle, and sends the initial motion state data to the second data processing module 102 for data arrangement to generate the motion state data of the vehicle.

[0190] Taking a vehicle as an example, the second data processing module 102 accesses a Controller Area Network (CAN) bus through a vehicle-mounted Ethernet, and the sensor collects the initial motion state data of the vehicle and transmits the data to the second data processing module 102 through the CAN bus. Various sensors can collect the initial motion state data of the vehicle in real time, wherein the initial motion state data includes at least one of yaw rate, speed, acceleration, and steering wheel angle.

[0191] Optionally, the second data processing module 102 collects the initial motion state data of the vehicle in the first 0.5 seconds of the current time through the sensor 400 in real time through a data collection frequency of 100 Hz.

[0192] The initial motion state data of the vehicle is processed by a preset timestamp alignment rule to eliminate the time deviation of each data in the initial motion state data. During the process of collecting the initial motion state data of the vehicle by the sensor, noise data may exist in the initial motion state data due to the different data collection methods between different sensors, the time difference of collecting data, etc., which affects the consistency of the initial motion state data.

[0193] Therefore, in the second data processing module 102, a sliding window weighted average filtering algorithm is used to smooth the original signal collected by the sensor, and a timestamp alignment mechanism is used to eliminate the time deviation of the multi-source initial motion state data obtained by different sensors. For example, the steering wheel angle data and the yaw rate in the initial motion state data are processed by a preset timestamp alignment rule to control the time error of the steering wheel angle data and the yaw rate within ±5ms, and the phase difference between the steering wheel angle data and the yaw rate is used to ensure the spatiotemporal consistency of the initial motion state data.

[0194] Then, in the second data processing module 102, the initial motion state data of the vehicle is input into a second-order dynamics model. The second-order dynamics model of the vehicle is based on a dynamic second-order state space model of the vehicle, and the historical initial motion state data of the previous 0.5 seconds is input to predict the motion trend of the vehicle within the next 100 milliseconds through a state space equation. For example, when the steering wheel angle rate of the vehicle is detected to exceed 2° / s, the second-order dynamics model calculates the expected yaw rate increment according to the wheelbase and the height of the center of mass of the vehicle, and dynamically adjusts the predicted motion state data of the vehicle, such as the current motion direction and / or the current motion speed of the vehicle, in combination with the road adhesion coefficient. The prediction result is converted into an optical performance parameter of a second light field region in the light field array, wherein the optical performance parameter includes data such as light wave propagation direction, speed and wavelength.

[0195] For example, for emergency braking or rapid lane changing scenarios, the second-order dynamics module introduces a reinforcement learning mechanism to improve the robustness of the prediction result by updating the dynamics model parameters online. The current motion speed prediction formula of the vehicle is:

[0196]

[0197] where t is the current time, Δt is the interval time, generally 0.5 seconds, ω pred (t+Δt) is the yaw rate, ω(t) is the initial yaw rate, is the angular acceleration in the interval time, and Δt is the interval time, is the second-order change of the angular acceleration in the interval time.

[0198] wherein, The steering wheel rotation rate and speed are coupled to estimate:

[0199]

[0200] K is a calibration coefficient, δ is the steering wheel rotation angle, and v is the speed.

[0201] In the vehicle control system of the present application, the data acquisition and control process adopts a multi-thread parallel architecture to optimize real-time performance. Each thread exchanges data through a lock-free ring buffer and compares the difference between the predicted light field and the actual somatosensory feedback in real time. The light field control data of the light field array is dynamically adjusted by a Proportion Integration Differentiation (PID) controller to form a closed-loop control circuit. The main thread runs the eye tracking algorithm at a frequency of 120Hz, completing one frame of image processing every 8.3ms; the secondary thread subscribes to CAN bus signals and calculates the motion state data of the vehicle through a Kalman filter algorithm, with a refresh cycle of 10ms; the third thread integrates the eye tracking results and motion state data to generate PWM control signals and sends them to the light field array through the SPI bus. All threads exchange data through a lock-free ring buffer, and the global clock synchronization error is less than 1ms. When it is detected that the passenger closes his eyes or the line of sight deviates from the light field for more than 3 seconds, the control system automatically turns off 80% of the Mini-LED units and enters a low-power standby mode, with a standby power consumption of 0.5W; in the tunnel entry and exit and other sudden light and dark scenes, the light field brightness of the background layer of the light field array is gradually adjusted within 500ms through an exponential fade curve to avoid the pupils from reacting to the sudden brightness change. All light field control data is stored in an encrypted Electrically Erasable Programmable Read-Only Memory (EEPROM), and the control system also supports remote updating of the built-in algorithm coefficients and / or adding new algorithms through Over-The-Air (OTA) technology, such as adjusting the light field control data for different regional driving habits.

[0202] The present application also provides an electronic device, which comprises:

[0203] a memory and a processor;

[0204] The memory stores a computer program, and the processor is configured to execute the computer program in the memory to perform the vehicle control method of the present application.

[0205] Figure 6 is a structural block diagram of the electronic device of the present application.

[0206] As shown in Figure 6 The electronic device 600 includes a processor 601 and a memory 603. The processor 601 and the memory 603 are connected, for example, via a bus 602. Optionally, the electronic device 600 can further include a transceiver 604. It should be noted that the transceiver 604 is not limited to one in actual applications, and the structure of the electronic device 600 does not constitute a limitation on the embodiments of the present application.

[0207] The processor 601 can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor 601 can implement or execute the various exemplary logical blocks, modules and circuits described in connection with the present disclosure. The processor 601 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0208] The bus 602 can include a path for transmitting information between the above-mentioned components. The bus 602 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 602 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 only one bus or one type of bus is represented by a thick line, but this does not mean that there is only one bus or one type of bus.

[0209] The memory 603 is used to store a computer program corresponding to the data processing method of the above-mentioned embodiments of the present application, which is controlled and executed by the processor 601. The processor 601 is used to execute the computer program stored in the memory 603 to realize the content shown in the foregoing method embodiments.

[0210] The electronic device 600 includes but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6The electronic device 600 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0211] The embodiments of the present application also provide a vehicle, which comprises the electronic device of the embodiments of the present application or the vehicle control system of the embodiments of the present application.

[0212] The vehicle can be a manned device such as a vehicle, a ship, a rail transit, an airplane, etc., and the present application does not make a specific limitation thereon. The vehicle can be a fuel vehicle, a plug-in hybrid vehicle, a new energy vehicle, etc., and the present application does not make a specific limitation thereon.

[0213] The vehicle of the present application can determine the light field control data of the light field array through the gaze data of the passenger and the motion state data of the vehicle, and control the light field array through the light field control data, so that the light field bright light area of the light field array tracks the gaze area of the passenger, and the optical performance parameters of the light field bright light area are adjusted in combination with the motion state data, effectively alleviating or preventing the motion sickness problem of the passenger during the unstable motion of the vehicle.

[0214] The embodiments of the present application also provide a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. The processor of the vehicle reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the vehicle perform the vehicle control method provided in various optional implementation manners in the above embodiments.

[0215] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a computer program, or by relevant hardware controlled by a computer program, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0216] According to an aspect of the present application, a computer readable storage medium is provided, which stores instructions thereon, the instructions, when executed by a processor, cause the processor to be configured to perform the vehicle control method described above.

[0217] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system, or a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, magnetic disks, CD-ROMs, optical storage devices, and the like) embodying computer program instructions.

[0218] The application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0219] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0220] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0221] In one typical configuration, the computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0222] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a belief store, and / or non-volatile memory, such as read-only memory (ROM), EPROM, EEPROM or flash memory. The memory is an example of computer readable media.

[0223] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0224] In the description of the present application, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0225] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can refer to the related description of other embodiments.

[0226] The embodiments, implementation manners and related technical features of the present application can be combined with each other without conflict.

[0227] The above is only the preferred embodiments of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application without departing from the technical solution content of the present application still falls within the scope of the technical solution of the present application.

Claims

1. A method for controlling a vehicle, characterized in that, include: The light field control data of the light field array is determined based on passenger gaze data and vehicle motion data. The light field array is controlled according to the light field control data.

2. The method according to claim 1, characterized in that, The light field control data includes first light field control data and second light field control data. Determining the light field control data of the light field array based on passenger gaze data and vehicle motion state data includes: The first light field control data is determined based on the passenger's gaze data; and / or, The second light field control data is determined based on the passenger's gaze data and the vehicle's motion state data.

3. The method according to claim 2, characterized in that, The gaze data includes visual focus data, and determining the first light field control data based on the passenger's gaze data includes: The first light field control data is determined based on the visual focus data.

4. The method according to claim 3, characterized in that, The light field array includes a first light field region, and controlling the light field array according to the light field control data includes: The first light field region is controlled according to the first light field control data.

5. The method according to claim 2, characterized in that, The gaze data includes peripheral vision data. Determining the second light field control data based on the passenger's gaze data and the vehicle's motion state data includes: The second light field control data is determined based on the peripheral vision data and the motion state data.

6. The method according to claim 5, characterized in that, The light field array includes a second light field region, and the second light field control data includes second light field region data and optical performance parameters. Determining the second light field control data based on the peripheral vision data and the motion state data includes: The second light field region data is determined based on the peripheral vision data; The optical performance parameters are determined based on the motion state data.

7. The method for obtaining motion state data according to claim 6, characterized in that, The light field array includes a second light field region, and controlling the light field array according to the light field control data includes: The second light field region is controlled based on the data from the second light field region. The optical performance of the second light field region is controlled according to the optical performance parameters.

8. The method according to claim 6, characterized in that, The motion status data includes at least one of the vehicle's current direction of motion and current speed; and / or, The optical performance parameters include at least one of the following: light wave propagation direction, light wave propagation speed, and light wave wavelength.

9. The method according to claim 7, characterized in that, The optical properties of the second light field region include at least one of the light wave propagation direction, light wave propagation speed, and light wave wavelength of the second light field region.

10. The method according to any one of claims 1 to 9, characterized in that, Before determining the optical performance parameters of the light field array based on passenger gaze data and vehicle motion data, the process includes: The gaze data is determined based on the passenger's eye image and a first distance; Wherein, the first distance is the distance from the passenger's eye to the target plane of the vehicle.

11. The method according to claim 10, characterized in that, Determining the gaze data based on the passenger's eye image and a first distance includes: Calculate the passenger's head pose with degrees of freedom based on the passenger's eye image; The deviation compensation for the passenger's head pose is performed based on the first distance to obtain the gaze data.

12. The method according to claim 11, characterized in that, The eye image includes the pupil outline and the cornea. Calculating the passenger's head pose based on the passenger's eye image includes: The passenger's eye gaze vector is calculated based on the pupil contour and the cornea, and the passenger's head pose with degrees of freedom is generated based on the gaze vector. The pose with degrees of freedom includes position parameters and / or orientation parameters in three-dimensional space. The position parameters include coordinate data in a three-dimensional coordinate system, and the orientation parameters include at least one of pitch angle, yaw angle, and roll angle.

13. The method according to claim 12, characterized in that, The step of calculating the gaze vector of the passenger's eyes based on the pupil contour and the cornea includes: The convolutional neural network model calculates the gaze vector of the passenger's eyes based on the pupil parameters of the pupil contour and the relative position of the corneal reflection point of the cornea; The pupil parameters of the pupil outline include the ratio of the major and minor axes of the pupil ellipse, the tilt angle, and the center coordinates.

14. The method according to claim 10, characterized in that, Before determining the gaze data based on the passenger's eye image and the first distance, the process includes: The eye image is subjected to distortion correction and coordinate system alignment in the camera coordinate system.

15. The method according to any one of claims 1 to 9, characterized in that, Before determining the light field control data of the light field array based on passenger gaze data and vehicle motion state data, the method further includes: The initial motion state data of the vehicle is processed by a preset filtering algorithm to obtain the motion state data of the vehicle.

16. The method according to claim 15, characterized in that, Also includes: The initial motion state data of the vehicle is processed by a preset timestamp alignment rule to eliminate time deviations in the data.

17. The method according to claim 15, characterized in that, The initial motion state data includes at least one of yaw rate, velocity, acceleration, and steering wheel angle.

18. A vehicle control system, characterized in that, include: A controller and an optical field array, wherein the controller and the optical field array are connected; The controller is used to determine the light field control data of the light field array based on the passenger's gaze data and the vehicle's motion state data. The light field array is controlled according to the light field control data.

19. The system according to claim 18, characterized in that, The controller includes: A first data processing module is configured to determine the passenger's gaze data based on the passenger's eye image and a first distance. Wherein, the first distance is the distance from the passenger's eye to the target plane of the vehicle.

20. The system according to claim 19, characterized in that, Also includes: An eye-tracking device, connected to the first data processing module, is used to acquire the passenger's eye image and the first distance.

21. The system according to claim 20, characterized in that, The eye-tracking module includes: Multiple cameras, wherein the multiple cameras are used to acquire images of the passenger's eyes; A distance sensor, which is used to acquire the first distance.

22. The system according to claim 18, characterized in that, The controller includes: The second data processing module is used to acquire the initial motion state data of the vehicle transmitted by the sensor, and to determine the motion state data of the vehicle based on the initial motion state data.

23. The system according to claim 18, characterized in that, The optical field array includes: LED array, the LED array including a microstructured diffusion film.

24. An electronic device, characterized in that, include: Memory and processor; The memory stores a computer program, and the processor is configured to run the computer program in the memory to perform the vehicle control method according to any one of claims 1 to 17.

25. A means of transportation, characterized in that, It includes the electronic device of claim 24, or the vehicle control system of any one of claims 18 to 23.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor using the vehicle control method as described in any one of claims 1 to 17.

27. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the vehicle control method according to any one of claims 1 to 17.

Images