Calibrating system, calibrating method and non-transitory computer readable storage medium thereof for calibrating predicted sight

Abstract

A calibration system in which the reference outcome calculation device generates eye position and reference line of sight angle at various historical time points. Filter Processing Device Filter processing to obtain the second predicted line of sight angle at these historical time points is performed based on the first predicted line of sight angle generated by the line of sight prediction device at each historical time point. The first predicted line of sight angle is obtained by the line-of-sight prediction device making a rough prediction based on the first image of the eye. The compensation value calculation device obtained offsets based on the second predicted line of sight angles and the reference line of sight angles at these historical time points. The configuration management device updates the second predicted line of sight angle and offset for these historical time points in the configuration file. The line-of-sight calibration device computes the mean offset of the second predicted line of sight angle according to the profile and compensates for the future predicted line of sight angle based on the mean offset.

Description

[Technical Field]

[0001] This disclosure relates to a correction mechanism, and more particularly to a correction system and method for correcting predicted line of sight. [Previous Technology]

[0002] Various emerging types of vehicles (such as electric vehicles) are typically equipped with driver assistance systems that use artificial intelligence to perform calculations and provide various types of driving assistance to the vehicle driver. These driver assistance systems may include a gaze prediction device, which predicts or estimates the driver's gaze direction to assess the driver's level of attention and determine whether the driver is in a dangerous driving state. When a dangerous driving state is determined, the driver assistance system can generate a warning signal to remind the driver or activate an automatic driver assistance function.

[0003] To ensure driving safety, the gaze prediction device of an assisted driving system must achieve a high level of prediction accuracy. However, different drivers may have individual physiological differences in their visual systems, which can lead to errors in the prediction results of the gaze prediction device. In addition, uncertainties in the vehicle's surrounding environment may also negatively affect the prediction accuracy of the gaze prediction device.

[0004] In response to the above issues, there is a need to provide an effective correction system that can correct errors in the vision prediction device of the driver assistance system, so as to improve the accuracy of the driver's vision direction prediction and thus improve driving safety. [Summary of the Invention]

[0005] According to one aspect of the present disclosure, a calibration system is provided, comprising a reference result calculation device, a filter processing device, a compensation value calculation device, a configuration management device and a line of sight calibration device. The reference result calculation device is used to generate the eye position of a user's eye at each of the plurality of historical time points, and to obtain a reference line of sight angle at each such historical time point based on the eye position and a target position associated with a target object, wherein the eye is a left or right eye of the user, the user is located inside a vehicle, and the target is located outside the vehicle. The filter processing apparatus is used to perform a filter processing to obtain a second predicted line of sight angle at each such historical time point based on a plurality of first predicted line of sight angles generated by a line of sight prediction device at those historical time points, wherein those first predicted line of sight angles are obtained by the line of sight prediction device based on a rough prediction of a plural number of first images associated with the eye. The compensating value calculation device is used to obtain a complex number of offsets from those second predicted line of sight angles and those reference line of sight angles at those historical time points. A configuration management device for updating in a set of profiles those second predicted line-of-sight angles and those offsets for those historical time points. The line-of-sight calibration device is used to calculate the average offset of one of those second predicted line-of-sight angles based on that profile, and to compensate for the future predicted line-of-sight angle accordingly.

[0006] According to another aspect of the present disclosure, a calibration method is provided comprising the following steps. By means of a one-reference result calculation device generates a one-eye position of a user's eye at each of the plurality of historical time points, and at each such historical time point obtains a reference line of sight angle based on the eye position and a target position associated with a target object, wherein the eye is a left or right eye of the user, the user is inside a vehicle, and the target is outside the vehicle. A second predicted line of sight angle is obtained at each of the historical time points by means of a filter processing device based on a complex number of first predicted line of sight angles generated by a line of sight prediction device at those historical time points, wherein the first predicted line of sight angles are obtained by the line of sight prediction device making a rough prediction based on a plural number of first images associated with the eye. By means of a compensation value calculation device a complex number of offsets is obtained based on those second predicted line-of-sight angles and those reference line-of-sight angles at those historical time points. Those second predicted line-of-sight angles and those offsets at those historical time points are updated in a set of state files by means of the configuration management device. By means of the first line of sight calibration device the configuration file calculates the average offset of one of those second predicted line of sight angles and compensates for the future predicted line of sight angle according to the mean offset.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, which stores a plurality of instructions that, when read by a controller, computer device or computer, enable the controller, computer device or computer to perform a correction method.

[0008] Other aspects and advantages of this disclosure will become apparent from the following figures, detailed description and claims.

Implementation Method

[0010] The technical terms used in this specification refer to those commonly used in the field of technical expertise. Where this specification provides explanations or definitions for certain terms, the interpretation of those terms shall be based on the explanations or definitions provided in this specification. Each embodiment disclosed herein has one or more technical features. Where feasible, those skilled in the art may selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

[0011] Figure 1 is a schematic diagram of the application environment of a correction system 1000 according to an embodiment of this disclosure. The correction system 1000 is installed on a vehicle 20; the vehicle 20 is driven by a user 10. The vehicle 20 is, for example, various types of vehicles, including four-wheeled cars, freight trucks, public transport buses, etc. Alternatively, the vehicle 20 may also be various types of aircraft or ships, including helicopters, small private jets, large public transport aircraft, yachts, cargo ships, etc. In the following embodiments, a four-wheeled car is used as an example of the vehicle 20.

[0012] The correction system 1000 is used to correct the line of sight of the user 10. The user 10 is the driver of the vehicle 20 and sits inside the vehicle 20. The user 10 can view a target 30 outside the vehicle 20 from inside the vehicle 20. The target 30 can be fixed or mobile, such as other vehicles, pedestrians, buildings, traffic signs, etc., around the vehicle 20. In this embodiment, the target 30 is another vehicle to the left front of the vehicle 20. More specifically, when the user 10 views the target 30, the user 10's eyes are focused on the target 30 according to the direction of the line of sight; the user 10's eyes are either the left eye 11 or the right eye 12. For example, the user 10's left eye 11 is focused on the target 30 according to the direction of the line of sight LS1, and the right eye 12 is focused on the target 30 according to the direction of the line of sight LS2. The correction system 1000 corrects the line of sight LS1 of the left eye 11 and the line of sight LS2 of the right eye 12 of the user 10 respectively. The embodiments described below only use the correction of the line of sight LS1 of the left eye 11 as an example (the technical solutions of the embodiments can also be applied to correct the line of sight LS2 of the right eye 12).

[0013] Figure 2 is a schematic diagram of the line of sight LS1 of the user 10's left eye 11. The line of sight LS1 can be specifically described using a coordinate system defined by the X-axis, Y-axis, and Z-axis. The eye position e1 of the left eye 11 serves as a reference point for the coordinate system, through which the X-axis, Y-axis, and Z-axis all pass. The Z-axis is, for example, substantially perpendicular to the ground; when the user 10 is seated inside the vehicle, the upper torso 13 of the user 10 is substantially parallel to the Z-axis. On the other hand, the X-axis and Y-axis are both orthogonal to the Z-axis; the X-axis is, for example, the frontal direction of the upper torso 13 of the user 10.

[0014] The line of sight LS1 of the left eye 11 extends from the eye position e1 to the target position p1 of the target object 30. The line of sight LS1 has an angle A1 with the X-axis, and the direction of the line of sight LS1 can be defined according to the angle A1. In three-dimensional space, the line of sight LS1 also has an angle with the Z-axis, but the angle between the line of sight LS1 and the Z-axis is not shown in the embodiment of Figure 2. The direction of the line of sight LS1 is illustrated only by way of example using the two-dimensional plane formed by the X-axis and Y-axis. In the embodiments described below, the correction system 1000 corrects the line of sight LS1 using the line of sight LS1 of the user 10's left eye 11 as an example.

[0015] Figure 3 is a block diagram of a correction system 1000 according to an embodiment of this disclosure. Figure 4 is a schematic diagram of the correction system 1000 installed on a carrier 20. Referring to both Figure 3 and Figure 4, the correction system 1000 includes a reference result calculation device 100, a filtering processing device 200, a compensation value calculation device 300, a configuration management device 400, and a line-of-sight correction device 500. The reference result calculation device 100 includes an eye position calculation unit 101 and a spatial conversion unit 103.

[0016] In one example, the correction system 1000 is a hardware circuit located inside the vehicle, such as a hardware processor, including, but not limited to, a digital signal processor (DSP), a central processing unit (CPU), and a micro control unit (MCU). The reference result calculation device 100, the filtering processing device 200, the compensation value calculation device 300, the configuration management device 400, and the line-of-sight correction device 500 are hardware circuit units inside the correction system 1000.

[0017] In another example, the correction system 1000 is implemented as a software module; the overall function of the correction system 1000 is implemented by executing the internal software code (which includes multiple instructions) of a hardware processor (e.g., digital signal processor, central processing unit, or microcontroller unit) or hardware device (e.g., controller, computer device, or computer). Furthermore, the reference result calculation device 100, filtering processing device 200, compensation value calculation device 300, configuration management device 400, and line-of-sight correction device 500 within the correction system 1000 are all software modules, and their respective functions are implemented through software code. The aforementioned software code can be stored on a non-transitory computer-readable storage medium; such non-transitory computer-readable storage media include various forms of non-transitory (non-volatile) memory, hard disks, USB flash drives, and other storage devices. The non-transitory computer-readable storage medium may be electrically connected to a hardware processor or hardware device, or the non-transitory computer-readable storage medium may be disposed within a hardware processor or hardware device. When the hardware processor or hardware device reads the aforementioned software code from the non-transitory computer-readable storage medium, the hardware processor or hardware device may execute the instructions in the software code to realize the respective functions of the reference result calculation device 100, the filtering processing device 200, the compensation value calculation device 300, the configuration management device 400, and the line-of-sight correction device 500.

[0018] The correction system 1000 operates in conjunction with the image capturing device 21, the sensing system 22, and the gaze prediction device 40. The image capturing device 21 is a camera installed inside the vehicle 20; for example, a rearview mirror 25 can be mounted on the inner side of the roof of the vehicle 20, and the image capturing device 21 can be mounted on the rearview mirror 25, with the lens of the image capturing device 21 facing the user 10. The image capturing device 21 captures an image of the user 10's face to generate a first image M1. The first image M1 includes the user 10's eyes, for example, the user's left eye 11. The image capturing device 21 generates the first image M1 and transmits the first image M1 to the eye position calculation unit 101 of the reference result calculation device 100.

[0019] The eye position calculation unit 101 performs image processing based on the first image M1 to calculate the eye position e1 of the left eye 11. Please also refer to Figure 2. The eye position e1 of the left eye 11 can be further represented by the coordinate positions {e1 x, e1 y, e1 z} of the X-axis, Y-axis and Z-axis.

[0020] Next, referring to Figure 4, the vehicle 20 is further equipped with a sensing system 22. The sensing system 22 is also an external component independent of the calibration system 1000. The sensing system 22 includes a sensing device 221 and a target position calculation unit 222. The sensing device 221 is, for example, a radar sensing device or a lidar sensing device. The sensing device 221 senses (e.g., measures the distance) the target object 30 outside the vehicle to obtain a sensing result S1 of the target object 30. The sensing device 221 transmits the sensing result S1 associated with the target object 30 to the target position calculation unit 222. The target position calculation unit 222 analyzes the sensing result S1 to calculate the target position p1 of the target object 30. Referring to Figure 2, the target position p1 of the target object 30 can be further represented by the coordinate positions {p1x, p1y, p1z} of the X-axis, Y-axis, and Z-axis.

[0021] Then, the spatial conversion unit 103 of the reference result calculation device 100 calculates the angle A1 between the line of sight LS1 of the left eye 11 and the X-axis based on the eye position e1 and the target position p1; and the spatial conversion unit 103 defines the direction of the line of sight LS1 according to the angle A1 between the line of sight LS1 and the X-axis. The reference result calculation device 100 uses the line of sight LS1 as the reference line of sight LS REF, and uses the angle A1 as the reference line of sight angle Δ REF.

[0022] As described above, the target position calculation unit 222 of the sensing system 22 analyzes the sensing result S1 obtained by the sensing device 221 to obtain the target position p1 of the target object 30. It has high accuracy and therefore high reliability, and can reflect the true position of the target object 30. In other words, the reference line of sight LS REF and the reference line of sight angle Δ REF obtained by the reference result calculation device 100 based on the above target position p1 have high reliability. The reference line of sight LS REF and the reference line of sight angle Δ REF can be collectively referred to as the reference result REF; the reference result REF is more likely to reflect the true state of the user 10's line of sight LS1 (i.e., "Ground Truth, GT"). Furthermore, the reference result calculation device 100 transmits the reference result REF to the compensation value calculation device 300.

[0023] The image capturing device 21 also transmits the first image M1, including the left eye 11, to the gaze prediction device 40 for gaze prediction. Just as the image capturing device 21 and the sensing system 22 are external devices separate from the correction system 1000 disclosed herein, the gaze prediction device 40 is also separate from the correction system 1000 disclosed herein. In the example of Figure 3, the gaze prediction device 40 and the image capturing device 21 are two independent devices. Alternatively, in other examples, the gaze prediction device 40 may be integrated into the image capturing device 21.

[0024] In operation, based on the first image M1 generated by the image capturing device 21, the gaze prediction device 40 performs gaze prediction on the gaze LS1 of the left eye 11 to generate a first gaze prediction result PD1 for the user 10's left eye 11. More specifically, the image capturing device 21 captures images of the user 10's left eye 11 multiple times at multiple time points within a specific time interval to generate multiple first images M1 respectively; and the gaze prediction device 40 performs multiple gaze predictions on the aforementioned multiple first images M1 to generate multiple first gaze prediction results PD1 corresponding to multiple time points.

[0025] The line-of-sight prediction made by the line-of-sight prediction device 40 is only a preliminary prediction, and the stability of the first line-of-sight prediction result PD1 may be low. Therefore, the line-of-sight prediction made by the line-of-sight prediction device 40 is called a "coarse prediction".

[0026] Please refer to Figure 5A, which is a schematic diagram of the first gaze prediction result PD1. As mentioned above, the gaze prediction device 40 performs multiple gaze predictions based on multiple first images M1 at multiple time points to generate multiple first gaze prediction results PD1; while Figure 5A only shows one of the multiple first prediction results PD1. The first gaze prediction result PD1 includes the predicted gaze direction and gaze angle of the left eye 11. Specifically, the first gaze prediction result PD1 includes the first predicted gaze LS PD1 and the first predicted gaze angle A PD1, as shown in equation (1): PD1 = {LS PD1, A PD1} (1)

[0027] The predicted gaze direction of the left eye 11 is represented by a first predicted gaze direction LS PD1, and the predicted gaze angle of the left eye 11 is represented by a first predicted gaze angle A PD1. The first predicted gaze angle A PD1 is, for example, the angle between the first predicted gaze direction LS PD1 and the X-axis. In other words, the multiple first gaze prediction results PD1 obtained by making multiple predictions at multiple time points may include multiple first predicted gaze directions LS PD1 and corresponding multiple first predicted gaze angles A PD1. The gaze prediction device 40 transmits the multiple first gaze prediction results PD1 to the filtering processing device 200.

[0028] The filtering processing device 200 performs filtering processing on a plurality of first line-of-sight prediction results PD1 to generate a second line-of-sight prediction result PD2. The filtering processing performed by the filtering processing device 200 is, for example, statistical processing of the plurality of first line-of-sight prediction results PD1, which can calculate statistical parameters of the plurality of first line-of-sight prediction results PD1. In this embodiment, the statistical processing is, for example, the probability distribution calculation of the plurality of first line-of-sight prediction results PD1, which can calculate the average value E_A PD1 of the plurality of first predicted line-of-sight angles A PD1, and the variation V_A PD1 of the probability distribution of these first predicted line-of-sight angles A PD1; and these first predicted line-of-sight angles A PD1 have a distribution range R0 of probability distribution. The distribution center of the distribution range R0 corresponds to the average value E_A PD1; and the size of the probability interval of the distribution range R0 depends on the variation V_A PD1.

[0029] Next, please refer to Figure 5B, which is a schematic diagram of the second line-of-sight prediction result PD2. The filtering processing device 200 uses the average value E_APD1 of a plurality of first predicted line-of-sight angles APD1 as the second predicted line-of-sight angle APD2; in other words, the filtering processing device 200 takes the average of a plurality of first predicted line-of-sight angles APD1 to obtain the second predicted line-of-sight angle APD2.

[0030] Furthermore, the line of sight corresponding to the second predicted line of sight angle A PD2 ​​is the second predicted line of sight LS PD2. The filtering processing device 200 integrates the second predicted line of sight angle A PD2 ​​and the second predicted line of sight LS PD2 into a second line of sight prediction result PD2. That is, the second line of sight prediction result PD2 includes the second predicted line of sight angle A PD2 ​​and the corresponding second predicted line of sight LS PD2, as shown in equation (2): PD2 = {LS PD2, A PD2} (2)

[0031] As mentioned above, the line-of-sight prediction device 40 performs a coarse prediction to generate multiple first line-of-sight prediction results PD1; while the second line-of-sight prediction result PD2 is a further filtering of the multiple first line-of-sight prediction results PD1. Therefore, the second line-of-sight prediction result PD2 has higher stability; and the filtering process performed by the filtering processing device 200 can be called "fine prediction".

[0032] The filtering processing device 200 transmits the second line-of-sight prediction result PD2 to the compensation value calculation device 300. On the other hand, the compensation value calculation device 300 receives the reference result REF generated by the spatial conversion unit 103 of the reference result calculation device 100. The compensation value calculation device 300 estimates the offset B based on the second line-of-sight prediction result PD2 and the reference result REF. The offset B is used as a compensation value, and its function is to compensate for the first line-of-sight prediction result PD1 generated by the line-of-sight prediction device 40 in the future.

[0033] Please refer to Figure 5C, which is a schematic diagram of the offset B. The offset B is the degree of deviation between the reference line LS REF in the reference result REF and the second predicted line LS PD2 in the second predicted line PD2 (that is, it represents the degree of error of the second predicted line PD2 compared to the reference result REF). In one example, the difference between the reference line angle A REF and the second predicted line angle A PD2 ​​is used as the offset B.

[0034] Furthermore, the compensation value calculation device 300 defines an effective region R1 based on the reference line-of-sight angle AREF, the offset B, and the variation V_APD1 (as mentioned above, the variation V_APD1 is the variation of the probability distribution of multiple first predicted line-of-sight angles APD1 at multiple time points).

[0035] Next, please refer to Figures 6A and 6B, which are schematic diagrams of the effective region R1. In Figure 6A, the effective region R1 is defined based on the reference center and the span C; the effective region R1 extends outward from the reference center with the span C. The reference center of the effective region R1 corresponds to the reference line of reference line LS REF of the reference line of reference angle A REF; and the span C of the effective region R1 is, for example, equal to the product of the variation V_A PD1 and the multiplier f plus the offset B. The multiplier f is, for example, a positive integer or a decimal, such as a positive integer "3". In other words, the angle obtained by subtracting the span C from the reference line of reference angle A REF is taken as the "starting angle A_s", and the angle obtained by adding the span C to the reference line of reference angle A REF is taken as the "ending angle A_e". Then the effective region R1 is the angular range covered by the "starting angle A_s" to the "ending angle A_e".

[0036] The specific meaning of the span C being equal to the sum of the product of the variation V_A PD1 and the multiplier f and the offset B is: to take into account both the variation V_A PD1 of the first predicted line-of-sight angle A PD1 and the offset B of the second predicted line-of-sight angle A PD2 ​​to evaluate the overall error of the line-of-sight prediction device 40; in other words, the span C can represent the overall error of the line-of-sight prediction device 40. Therefore, the effective area R1 expanded according to the span C can represent the possible angular range of the target position p1 of the target object 30. Each angle within the effective area R1 may be the angle of the target position p1. If more than two targets are detected within the effective area R1 (i.e., the effective area R1 includes other targets 32 besides the target object 30; the target position p2 of the target object 32 falls within the angular range of the effective area R1), then the second line-of-sight prediction result PD2 obtained by the line-of-sight prediction device 40 and the collaborative filtering processing device 200 in this prediction is considered invalid.

[0037] Conversely, if only a single target object 30 exists within the effective area R1, the second line-of-sight prediction result PD2 generated this time is considered valid, and its corresponding offset B (i.e., the difference between the reference line-of-sight angle AREF and the second predicted line-of-sight angle APD2 generated this time) is also considered valid. The compensation value calculation device 300 transmits the offset B, which is determined to be valid this time, to the configuration management device 400. Furthermore, the configuration management device 400 updates this valid offset B in the configuration file.

[0038] Figure 6A illustrates the coverage of the effective region R1 using only a two-dimensional plane along the X-Y axis as an example. Please refer to Figure 6B, which illustrates the coverage of the effective region R1 xz in a three-dimensional space along the X-Y-Z axis. In the three-dimensional space along the X-Y-Z axis, the reference line of sight LS REF points to the reference center of the effective region R1 xz (i.e., the reference center of the effective region R1 xz is the target position p1 of the target object 30). Furthermore, the effective region R1 xz has two dimensions of expansion, namely expansion Cx and expansion Cz. Among them, expansion Cx is the expansion C shown in Figure 6A, which is the angular range of the effective region R1 in the two-dimensional plane along the X-Y axis. And expansion Cz is the angular range of the effective region R1 xz in the two-dimensional plane along the Y-Z axis. The definition of span Cz is similar to that of span Cx; span Cz is the sum of the offset and variation multiples of the angle between the reference line of sight LS REF and the Z-axis (not shown in Figure 6B). Regarding the validity of the second line of sight prediction result PD2, if the target position p2 of another target object 32 besides target object 30 falls within the effective area R1 xz, then the second line of sight prediction result PD2 obtained by the line of sight prediction device 40 and the filtering processing device 200 in this prediction is deemed invalid. Conversely, if only the target position p1 of target object 30 is detected within the effective area R1 xz, then the second line of sight prediction result PD2 obtained in this prediction is deemed valid. The configuration management device 400 only updates the offset B corresponding to the second line of sight prediction result PD2 that is deemed valid in the configuration file.

[0039] In summary, based on the embodiments shown in Figures 3, 5A-5C, 6A, and 6B, when the user 10's left eye 11 gazes at the target object 30 from a certain viewing angle, the image capturing device 21 generates a first image M1 including the left eye 11, and the gaze prediction device 40 predicts the gaze LS1 of the left eye 11 based on the first image M1 to obtain a first gaze prediction result PD1. The correction system 1000 can decide whether to include the first gaze prediction result PD1 in the compensation calculation process according to preset conditions. The preset conditions are, for example, the complexity of the environment outside the vehicle; if the complexity of the environment outside the vehicle is high, the correction system 1000 will not include the first gaze prediction result PD1 in the compensation calculation process. In one example, the correction system 1000 can refer to the sensing result S1 generated by the sensing device 221 to determine the complexity of the environment outside the vehicle. If the sensing result S1 indicates that the number of objects in the environment outside the vehicle is large (e.g., the number of objects is greater than the upper limit), the correction system 1000 will not include the first gaze prediction result PD1 in the compensation calculation process. If the sensing result S1 indicates that the number of objects in the environment outside the vehicle is less than the upper limit, then the first line-of-sight prediction result PD1 is included in the compensation calculation process.

[0040] Furthermore, the filtering processing device 200 of the correction system 1000 performs filtering processing based on the adopted first line-of-sight prediction result PD1 to obtain a second line-of-sight prediction result PD2 (including the second predicted line of sight LS PD2 and the second predicted line-of-sight angle A PD2). On the other hand, the reference result calculation device 100 of the correction system 1000 calculates the reference result REF of the line of sight of the left eye 11 based on the first image M1 and the target position p1 of the target object 30 (including the reference line of sight LS REF and the reference line-of-sight angle A REF). Then, the compensation value calculation device 300 of the correction system 1000 calculates the offset B based on the second line-of-sight prediction result PD2 and the reference result REF. Furthermore, the compensation value calculation device 300 can determine whether the second line-of-sight prediction result PD2 is valid based on the number of target objects within the effective area R1. If the second line-of-sight prediction result PD2 is valid, the correction system 1000 can adopt the offset B corresponding to the valid second line-of-sight prediction result PD2. The compensation value calculation device 300 transmits the adopted offset B to the configuration management device 400, and the configuration management device 400 updates the adopted offset B in the configuration file. Similarly, the user 10's left eye 11 can look at other target objects from another line-of-sight angle, and the compensation value calculation device 300 calculates the offset B corresponding to this line-of-sight angle. Furthermore, the compensation value calculation device 300 can calculate the offset B for each of multiple line-of-sight angles within a predetermined angle range and update these offset Bs in the configuration file of the configuration management device 400. The configuration management device 400 continuously records the offset B corresponding to each line-of-sight angle, as detailed below.

[0041] Please refer to Figure 7A, which is a schematic diagram of the compensation value calculation device 300 calculating the offset B within a predetermined angle range and setting the correction point. The compensation value calculation device 300 calculates the offset B corresponding to the second line-of-sight prediction result PD2 at different angles. For example, the offset B corresponding to the second line-of-sight prediction result PD2 is calculated at 20 degrees, 35 degrees, 50 degrees, 65 degrees and 80 degrees respectively: the offset B calculated at 20 degrees is "3.3 degrees", the offset B calculated at 35 degrees is "3.5 degrees", the offset B calculated at 50 degrees is "3.1 degrees", the offset B calculated at 65 degrees is "2.8 degrees" and the offset B calculated at 80 degrees is "2.9 degrees". The offsets B of "3.3 degrees", "3.5 degrees", "3.1 degrees", "2.8 degrees" and "2.9 degrees" mentioned above are discretized in a binning manner and set as correction points.

[0042] According to the above mechanism, the compensation value calculation device 300 sets multiple correction points, for example, five correction points, which correspond to 20 degrees, 35 degrees, 50 degrees, 65 degrees and 80 degrees respectively. When the first predicted viewing angle A PD1 predicted by the viewing prediction device 40 is equal to one of the five correction points of 20 degrees, 35 degrees, 50 degrees, 65 degrees and 80 degrees, the viewing correction device 500 directly compensates for the future first predicted viewing angle A PD1 by using the average value of the offset B of the correction points corresponding to the angles equal to the first predicted viewing angle A PD1 recorded in the configuration file (called "average offset B'"). That is, the average offset B' of the correction points corresponding to the angles equal to the first predicted viewing angle A PD1 is added to the first predicted viewing angle A PD1 to obtain the corrected viewing prediction result PD1b.

[0043] If the predicted first viewing angle A PD1 is not equal to any of the five correction points of 20 degrees, 35 degrees, 50 degrees, 65 degrees, and 80 degrees, the viewing correction device 500 compensates for the average offset corresponding to the interval into which the predicted viewing angle A PD1 falls. An interval is, for example, the range between two adjacent correction points. For instance, the first interval SEG1 is the range between the two correction points of 20 degrees and 35 degrees, the second interval SEG2 is the range between the two correction points of 35 degrees and 50 degrees, the third interval SEG3 is the range between the two correction points of 50 degrees and 65 degrees, and the fourth interval SEG4 is the range between the two correction points of 65 degrees and 80 degrees. In one example, all predicted viewing angles A PD1 falling within the same interval are compensated for with the same average offset to obtain the corrected viewing prediction result PD1b. Please refer to Figure 7B, which is a schematic diagram of the correction of the first predicted viewing angle A PD1 according to the average offset of the corresponding interval. For example, a first predicted viewing angle A PD1 is 30 degrees, falling within the range of the first interval SEG1. Any first predicted viewing angle A PD1 falling within the first interval SEG1 is compensated with the same average offset B'_1. The average offset B "3.3 degrees" of the correction point of 20 degrees at one end of the first interval SEG1 is used as the average offset B'_1 corresponding to the first interval SEG1. Adding the average offset B'_1 "3.3 degrees" to the first predicted viewing angle A PD1 results in the compensated viewing angle C_A "33.3 degrees". According to the above method, compensating for different angles of the first predicted viewing angle A PD1 within the same interval with the same average offset simplifies the calculation complexity and reduces the amount of data.

[0044] Similarly, the average offset B' (3.5 degrees) of the correction point at one end of the second interval SEG2 is taken as the offset B'_2 corresponding to the second interval SEG2; any first predicted viewing angle A PD1 falling into the second interval SEG2 is compensated with the same average offset B'_2 (the first predicted viewing angle A PD1 is added to the average offset B'_2). By analogy, any first predicted viewing angle A PD1 falling into the third interval SEG3 is compensated with the same average offset B'_3, and any first predicted viewing angle A PD1 falling into the fourth interval SEG4 is compensated with the same average offset B'_4.

[0045] In another example, the gaze correction device 500 interpolates using the average offset B' of the two closest correction points to the first predicted gaze angle A PD1, and compensates for the first predicted gaze angle A PD1 with the interpolated value. Referring to Figure 7A, taking linear interpolation as an example, if the first predicted gaze angle A PD1 is 25 degrees, then the compensation is performed using the linear interpolation value of the average offset B' "3.3 degrees" of the closest correction point at 20 degrees and the average offset B' "3.5 degrees" of the correction point at 35 degrees. The linear interpolation value of the average offset B' "3.3 degrees" and the average offset B' "3.5 degrees" is equal to "the sum of 3.3 degrees multiplied by 2 / 3 and 3.5 degrees multiplied by 1 / 3".

[0046] In another example, the line-of-sight correction device 500 compensates for the average offset B' of the correction point corresponding to the angle closest to the first predicted line-of-sight angle A PD1.

[0047] In the embodiments shown in Figures 7A and 7B, the interval between the plurality of correction points is, for example, 15 degrees. Also, please refer to Figure 7C, which is another schematic diagram of the compensation value calculation device 300 setting correction points; in the embodiment shown in Figure 7C, the interval between the plurality of correction points is, for example, 1 degree (the angle of the correction points is, for example, 20 degrees, 21 degrees, 22 degrees, etc.).

[0048] Next, please refer to Figure 8, which is a schematic diagram of the detailed operation of the correction system 1000 in Figure 3. In the example in Figure 8, each signal has a parameter "1", parameter "2", ..., parameter "n" and parameter "n+1", which represent time points t1, t2, ..., t(n) and t(n+1), respectively. Each time point from t1 to t(n) can represent a "historical time point" of the operation of the correction system 1000, and time point t(n+1) represents a "future time point" of the operation of the correction system 1000.

[0049] The image capturing device 21 generates multiple first images M1(1,2,…,n+1) at time points t1 to t(n+1). Furthermore, the sensing device 221 of the sensing system 22 generates multiple sensing results S1(1,2,…,n+1) at time points t1 to t(n+1). The target position calculation unit 222 of the sensing system 22 generates multiple target positions p1(1,2,…,n+1) associated with the target object 30 based on the sensing results S1(1,2,…,n+1).

[0050] At historical time points t1 to t(n), the reference result calculation device 100 of the correction system 1000 operates, wherein the eye position calculation unit 101 calculates multiple eye positions e(1,2,…,n) based on the first image M1(1,2,…,n) at historical time points t1 to t(n). Furthermore, the spatial conversion unit 103 calculates the reference result REF(1,2,…,n) based on the target position p1(1,2,…,n) at historical time points t1 to t(n).

[0051] On the other hand, the gaze prediction device 40 generates multiple first gaze prediction results PD1(1,2,…,n) based on the first images M1(1,2,…,n) at historical time points t1~t(n). The filtering processing device 200 performs filtering processing based on the first gaze prediction results PD1(1,2,…,n) to generate multiple second gaze prediction results PD2(1,2,…,n).

[0052] At historical time points t1 to t(n), the compensation value calculation device 300 calculates multiple offsets B(A PD2) based on the reference result REF(1,2,…,n) and the second line-of-sight prediction result PD2(1,2,…,n). The second predicted line-of-sight angle A PD2 ​​associated with the offset B(A PD2) corresponds to the angle of the correction point. The compensation value calculation device 300 can determine whether the offset B(A PD2) is valid; if valid, the offset B(A PD1) is transmitted to the configuration management device 400 and updated in the configuration file. In the configuration file, the first predicted line-of-sight angle A PD2 ​​associated with the offset B(A PD2) is established as the correction point.

[0053] At a future time point t(n+1), the gaze prediction device 40 generates a first gaze prediction result PD1(n+1) based on the first image M1(n+1). The first gaze prediction result PD1(n+1) includes a first predicted gaze angle A PD1(n+1), which can be referred to as the "future predicted gaze angle". Furthermore, the gaze correction device 500 of the correction system 1000 operates at the future time point t(n+1) and obtains the average offset B'(A PD2) of the correction point corresponding to the second predicted gaze angle A PD2 ​​at the same angle as the first gaze prediction result PD1(n+1) from the configuration file of the configuration management device 400. The gaze correction device 500 compensates for the first predicted gaze angle A PD1(n+1) (i.e., the future predicted gaze angle) generated by the gaze prediction device 40 based on the average offset B'(A PD2) to generate a corrected gaze prediction result PD1b(n+1).

[0054] Figures 9A and 9B are flowcharts of a correction method according to an embodiment of the present disclosure. The correction method of this embodiment is implemented, for example, by the correction system 1000 in Figure 3. Referring first to Figure 9A, step S900 is performed: the reference result calculation device 100 receives a first image M1 containing the eye (e.g., left eye 11) of the user 10 generated by the image capturing device 21, and receives a target position p1 generated by the sensing system 22.

[0055] Next, step S902 is performed: the first image M1 of the eye is analyzed by the eye position calculation unit 101 to obtain the eye position e1.

[0056] Next, step S904 is performed: the spatial conversion unit 103 calculates the reference line of sight angle ΔREF based on the eye position e1 and the target position p1.

[0057] Next, step S906 is executed: the line-of-sight prediction device 40 performs a coarse prediction based on the first image M1 to obtain a plurality of first line-of-sight prediction results PD1, which include a plurality of first predicted line-of-sight angles A PD1.

[0058] Next, step S908 is executed: the filtering processing device 200 performs filtering processing on the multiple first line-of-sight prediction results PD1 to obtain a second line-of-sight prediction result PD2. The second line-of-sight prediction result PD2 includes a second predicted line-of-sight angle A PD2. The filtering processing is, for example, performing statistical processing on the multiple first predicted line-of-sight angles A PD1 to calculate the average value E_A PD1 and the variation V_A PD1 of the first predicted line-of-sight angles A PD1. And, the average value E_A PD1 is used as the second predicted line-of-sight angle A PD2.

[0059] Next, please refer to Figure 9B and perform step S910: the compensation value calculation device 300 calculates the offset B based on the reference viewing angle A REF and the second predicted viewing angle A PD2.

[0060] Next, step S912 is executed: the effective area R1 is defined based on the reference line angle A REF, the offset B and the variation V_A PD1.

[0061] Next, step S914 is executed: determine whether there are other targets besides target 30 in the effective area R1. If the determination result is "no" (that is, there is only a single target 30 in the effective area R1), then step S916 is executed: determine that the current offset B is valid, and set the valid offset B as the correction point.

[0062] Then, step S918 is executed: the configuration management device 400 updates the offset B, which is valid and set as the correction point, in the configuration file.

[0063] Next, step S920 is executed: the line of sight correction device 500 calculates the average offset B' of the correction point corresponding to the first predicted line of sight angle A PD1, and compensates the line of sight prediction device 40 for the first predicted line of sight angle A PD1 currently predicted by the line of sight prediction device 40 with the average offset B'.

[0064] Although this disclosure has been described in detail above with reference to preferred embodiments and examples, it is understood that these examples are intended to be illustrative rather than limiting. It is anticipated that various modifications and combinations will be conceived by those skilled in the art, and such modifications and combinations fall within the spirit of this disclosure and the scope of the appended claims. [Simplified Explanation of the Diagram]

[0009] Figure 1 is a schematic diagram of the application environment of the correction system 1000 according to an embodiment of the present disclosure. Figure 2 is a schematic diagram of the line of sight LS1 of the left eye 11 of the user 10. Figure 3 is a block diagram of the correction system 1000 according to an embodiment of the present disclosure. Figure 4 is a schematic diagram of the correction system 1000 installed on the vehicle 20. Figure 5A is a schematic diagram of the first line of sight prediction result PD1. Figure 5B is a schematic diagram of the second line of sight prediction result PD2. Figure 5C is a schematic diagram of the offset B. Figures 6A and 6B are schematic diagrams of the effective area R1. Figure 7A is a schematic diagram of the compensation value calculation device 300 calculating the offset B within a predetermined angle range and setting the correction point. Figure 7B is a schematic diagram of the first predicted line of sight angle A PD1 being corrected according to the offset of the corresponding interval. Figure 7C is another schematic diagram of the compensation value calculation device 300 setting the correction point. Figure 8 is a schematic diagram of the detailed operation of the correction system 1000 in Figure 3. Figures 9A and 9B are flowcharts of the correction method according to an embodiment of the present disclosure.

Claims

1. A calibration system, comprising: a reference result calculation apparatus for generating at each of a plurality of historical time points the eye position of a user's eye part, and obtaining a reference line of sight angle at each such historical time point based on the eye position and a target position associated with a target object, wherein the eye is a left or right eye of the user, the user is inside a vehicle, and the target is outside the vehicle; a filter processing device for obtaining a second predicted line of sight angle at each such historical time point based on a plurality of first predicted line of sight angles generated by a line of sight prediction device at those historical time points, wherein the first predicted line of sight angles are obtained by the line of sight prediction device based on a rough prediction of a plural number of first images associated with the eye; a compensating value calculation apparatus for obtaining a complex number of offsets from those second predicted line-of-sight angles and those reference line-of-sight angles at those historical time points, and defines an effective region based on a variation of the reference line-of-sight angle, such offset, and those first predicted line-of-sight angles, wherein a datum center of the valid region corresponds to that reference line-of-sight angle at that target location of the target object; a set of state management devices for updating in a set of profiles with those offsets for those second predicted line of sight angles at those historical time points; 2. The calibration system of claim 1, wherein those first images associated with the eye are generated for an image retrieval device disposed inside the vehicle.

3. The correction system as described in claim 2, wherein the reference result calculation device includes: an eye position calculation unit for analyzing those first images associated with the eye to obtain the eye position; and a spatial transformation unit for calculating that reference line of sight angle based on that eye location and that target location.

4. A calibration system such as request 1, wherein the filter processing device performs the filter processing to obtain a mean value and the amount of variation of those first predicted line-of-sight angles, and the second predicted line-of-sight angle is equal to the average.

5. Such as the calibration system of claim 4, wherein the compensation value calculation device determines whether there is another target object other than the target object in the effective area.

6. Such as the calibration system of claim 5, where one spread of the effective region is equal to the product of the variable and the doubling rate summed to the offset.

7. As in the calibration system of claim 5, wherein the compensation value calculation device judges that the second predicted line of sight angle and the corresponding offset as valid when only the target object is present in the valid region.

8. such as the calibration system of Requisition 7, where the configuration management device is updating a valid such offset in the configuration file.

9. A calibration system such as request 8, wherein the compensation value computing device establishes a calibration point based on the first predicted line of sight angle associated with the offset.

10. A calibration system such as request 9, wherein the line-of-sight calibration device compensates for the future forecast line-of-sight angle with such average offset corresponding to the correction point at an angle equal to the future forecast line-of-sight angle.

11. A method of calibration comprising the following steps: By means of a reference result calculation device generates the eye position of a user's eye part at each of a plurality of historical points in time, , and at each such historical time point obtains a reference line of sight angle based on the eye position and a target position associated with a target object, wherein the eye is a left or right eye of the user, the user is located inside the object, and carries the target; A second predicted line of sight angle is obtained at each such historical time point by means of a filter processing device based on a complex number of first predicted line of sight angles generated by a line of sight prediction device at those historical time points, wherein the first predicted line of sight angles are obtained by the line of sight prediction device based on a rough prediction of a plural number of first images associated with the eye; A complex number of offsets is obtained by means of a compensating value calculating device based on those second predicted line-of-sight angles and those reference line-of-sight angles at those historical time points, and a valid region is defined based on the reference line-of-sight angle, the offset and one variation of the first predicted line-of-sight angle, wherein a datum center of the valid region corresponds to that reference line-of-sight angle of that target location of the target object; By means of a line-of-sight calibration device the average offset of one of those second predicted line-of-sight angles is calculated according to the profile, and a future predicted line-of-sight angle is compensated according to the average offset.

12. The calibration method of claim 11, wherein those first images associated with the eye are generated for an image retrieval device disposed inside the vehicle.

13. The correction method as described in claim 12, wherein the step of generating the eye position and the target position using the reference result calculation device includes: those first images associated with the eye are analyzed by means of the eye position calculation unit of the reference result calculation device to obtain the eye position; and by means of the one-spatial transformation unit of the reference result calculation device to calculate the reference line of sight angle based on the eye position and the target position.

14. The correction method as described in claim 11, wherein the steps of performing the filtration process by the filtration device include: A mean value of those first predicted line-of-sight angles and the amount of variation are calculated by means of the filtering processing device; 15. The correction method as described in claim 14 further includes: The compensatory value calculation device is used to determine whether there is another target object other than the target object in the effective area.

16. Such as the correction method of claim 15, wherein one spread of the effective region is equal to the product of the variation and the doubling rate summed to the offset.

17. As in the calibration method of claim 15, wherein the second predicted line of sight angle and the corresponding offset are judged to be valid when only the target object is present in the valid region.

18. The correction method as described in claim 17 further includes: By resorting to the configuration, the management device updates the valid that offset in the configuration file.

19. The correction method as described in claim 18 further includes: The computational device by means of the compensation value is established as a correction point based on that first predicted line of sight angle associated with the offset.

20. The correction method as described in claim 19 further includes: By means of the line-of-sight correction device compensating the future forecast line-of-sight angle with such average offset corresponding to the correction point at an angle equal to the future forecast line-of-sight angle.

21. A non-transitory computer-readable storage medium storing a plurality of instructions that, when read by a controller, a computer device, or a computer, execute the correction method described in any one of claims 11 to 20.

Images