Correction system for correcting predictive gaze, correction method, and non-transitory

Abstract

The invention relates to a correction system. A reference result calculation device in the correction system generates an eye position and a reference line-of-sight angle at each historical time point. The filtering processing device carries out filtering processing according to the first predicted sight line angle generated by the sight line prediction device at each historical time point so as to obtain a second predicted sight line angle at the historical time points. The first predicted line-of-sight angle is obtained by performing rough prediction by the line-of-sight prediction device according to a first image of the eye. The compensation value calculation device obtains an offset according to the second predicted line-of-sight angle and the reference line-of-sight angle at the historical time points. The configuration management device updates the second predicted sight angles and the offsets of the historical time points in the configuration file. The line-of-sight correction device calculates the average offset of the second predicted line-of-sight angle according to the configuration file, and compensates the future predicted line-of-sight angle according to the average offset.

Description

TECHNICAL FIELD

[0001] The present application relates to a correction mechanism, in particular to a correction system and a correction method for correcting a predicted line of sight. BACKGROUND

[0002] Various emerging types of vehicles, such as electric vehicles, are usually equipped with an assisted driving system to provide various types of driving assistance to the driver of the vehicle through artificial intelligence calculation. Among them, the assisted driving system can include a line of sight prediction device for predicting or estimating the line of sight direction of the driver, so as to evaluate the degree of attention of the driver to determine whether the driver is in a dangerous driving state. When it is determined that the driver is in a dangerous driving state, the assisted driving system can generate a warning signal to remind the driver or start an automatic assisted driving function.

[0003] In order to ensure driving safety, the line of sight prediction device of the assisted driving system must achieve high prediction accuracy. However, there may be individual physiological differences in the eye vision system of different drivers, which may cause errors in the prediction results of the line of sight prediction device. In addition, uncertain factors in the environment around the vehicle may also negatively affect the prediction accuracy of the line of sight prediction device.

[0004] In response to the above issues, it is necessary to provide an effective correction system that can correct errors occurring in the line of sight prediction device of the assisted driving system to improve the prediction accuracy of the line of sight direction of the driver and thus improve driving safety. SUMMARY

[0005] According to one aspect of the present invention, a correction system is provided, comprising a reference result calculation device, a filtering processing device, a compensation value calculation device, a configuration management device, and a gaze correction device. The reference result calculation device is used to generate an eye position of a user's eye at each of a plurality of historical time points, and to obtain a reference gaze angle at each of the historical time points based on the eye position and a target position associated with a target object, wherein the eye is the user's left or right eye, the user is located inside a vehicle, and the target object is located outside the vehicle. The filtering processing device is used to perform a filtering process on a plurality of first predicted gaze angles generated by a gaze prediction device at the historical time points to obtain a second predicted gaze angle at each of the historical time points, wherein the first predicted gaze angles are obtained by the gaze prediction device through coarse prediction based on a plurality of first images associated with the eye. The compensation value calculation device is used to obtain a plurality of offsets based on the second predicted gaze angles and the reference gaze angles at the historical time points. The configuration management device is used to update the second predicted gaze angles and the offsets at the historical time points in a configuration file. The line-of-sight correction device is used to calculate an average offset of the second predicted line-of-sight angles based on the configuration file, and to compensate for the future predicted line-of-sight angles based on the average offset.

[0006] According to another aspect of the present invention, a correction method is provided, comprising the following steps: A reference result calculation device generates an eye position of a user's eye at each of a plurality of historical time points, and at each of the historical time points, obtains a reference gaze angle based on the eye position and a target position associated with a target object, wherein the eye is the user's left or right eye, the user is located inside a vehicle, and the target object is located outside the vehicle. A filtering processing device performs a filtering process on a plurality of first predicted gaze angles generated by a gaze prediction device at the historical time points to obtain a second predicted gaze angle at each of the historical time points, wherein the first predicted gaze angles are obtained by the gaze prediction device through coarse prediction based on a plurality of first images associated with the eye. A compensation value calculation device obtains a plurality of offsets based on the second predicted gaze angles and reference gaze angles at the historical time points. A configuration management device updates the second predicted gaze angles and offsets at the historical time points in a configuration file. The configuration file uses a line-of-sight correction device to calculate an average offset of the second predicted line-of-sight angles, and compensates for the future predicted line-of-sight angles based on the average offset.

[0007] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, which stores a plurality of instructions, wherein when the instructions are read by a controller, computer device or computer, the controller, computer device or computer executes a correction method.

[0008] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the present invention. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the application environment of a calibration system 1000 according to an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of the line of sight LS1 of the left eye 11 of user 10.

[0011] Figure 3 This is a block diagram of a correction system 1000 according to an embodiment of the present invention.

[0012] Figure 4 A schematic diagram of the calibration system 1000 installed on the vehicle 20.

[0013] Figure 5A This is a schematic diagram of the first-line prediction result PD1.

[0014] Figure 5B This is a schematic diagram of the second line of sight prediction result PD2.

[0015] Figure 5C This is a schematic diagram of the offset B.

[0016] Figure 6A and 6B This is a schematic diagram of the effective region R1.

[0017] Figure 7A A schematic diagram showing how the compensation value calculation device 300 calculates the offset B within a predetermined angle range and sets the correction point.

[0018] Figure 7B This is a schematic diagram showing how the first predicted line-of-sight angle APD1 is corrected based on the offset of the corresponding interval.

[0019] Figure 7C Another schematic diagram for setting the correction point for the compensation value calculation device 300.

[0020] Figure 8 for Figure 3 A schematic diagram showing the detailed operation of the calibration system 1000.

[0021] Figure 9A , 9B This is a flowchart of a correction method according to an embodiment of the present invention.

[0022] Figure label:

[0023] 1000: Calibration System

[0024] 100: Reference Result Calculation Device

[0025] 101: Eye position calculation unit

[0026] 103: Spatial Conversion Unit

[0027] 200: Filtration treatment device

[0028] 300: Compensation value calculation device

[0029] 400: Configuration Management Device

[0030] 500: Visual correction device

[0031] 10: Users

[0032] 11: Left eye

[0033] 12: Right eye

[0034] 13: Upper body / torso

[0035] 20: Vehicles

[0036] 21: Image acquisition device

[0037] 22: Sensing System

[0038] 221: Sensing device

[0039] 222: Target position calculation unit

[0040] 25: Rearview Mirror

[0041] 30: Target object

[0042] 32: Other targets

[0043] 40: Line of sight prediction device X, Y, Z axes

[0044] M1: First Image

[0045] S1: Sensing Results

[0046] REF: Reference Results

[0047] PD1: First line of sight prediction result; PD1b: Corrected line of sight prediction result; PD2: Second line of sight prediction result (B, B(A)). PD1 ),B(A PD2 ): Offset B', B'(A PD2 ): Average offset B'_1, B'_2, B'_3, B'_4: Average offset e1: Eye position

[0048] {e1 x e1 y e1 z}: Coordinate positions p1, p2: Target position

[0049] {p1 x p1 y p1 z}: Coordinate positions LS1, LS2: Line of sight

[0050] A1: Angle

[0051] LS REF Reference line of sight A REF Reference line of sight (LS) PD1 First predictive line of sight A PD1 First predicted line of sight angle LS PD2 Second predictive line of sight A PD2 Second predicted line-of-sight angle R0: Distribution range

[0052] E_A PD1 :average value

[0053] V_A PD1 :variance f:multiplier

[0054] R1,R1 xz : Valid area

[0055] C,C x C z : Exhibition Scale

[0056] A_e: Termination Angle

[0057] A_s: Starting angle

[0058] SEG1: First Interval

[0059] SEG2: Second Interval

[0060] SEG3: Third Interval

[0061] SEG4: Fourth Interval

[0062] C_A: Compensated viewing angle

[0063] S900~S920: Procedures Detailed Implementation

[0064] The structural and working principles of the present invention will be described in detail below with reference to the accompanying drawings:

[0065] The technical terms used in this specification refer to those commonly used in the field. Where this specification provides explanations or definitions for certain terms, the interpretation of those terms shall be based on the explanations or definitions provided in this specification. Each embodiment of the present invention has one or more technical features. Where feasible, those skilled in the art may selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

[0066] Figure 1 This is a schematic diagram illustrating the application environment of a correction system 1000 according to an embodiment of the present invention. The correction system 1000 is installed in a vehicle 20, which is driven by a user 10. The vehicle 20 is, for example, various types of vehicles, including four-wheeled caravans, freight trucks, public transport buses, etc. Alternatively, the vehicle 20 may also be various types of aircraft or ships, including helicopters, small private jets, large public transport aircraft, yachts, cargo ships, etc. In the following embodiments, a four-wheeled caravan is used as an example of the vehicle 20.

[0067] The correction system 1000 is used to correct the line of sight of user 10. User 10 is the driver of vehicle 20 and sits inside vehicle 20. User 10 can view a target 30 outside vehicle 20 from inside vehicle 20. Target 30 can be fixed or mobile, such as other vehicles, pedestrians, buildings, traffic signs, etc., around vehicle 20. In this embodiment, target 30 is another vehicle to the left front of vehicle 20. More specifically, when user 10 views target 30, user 10's eyes are focused on target 30 according to the direction of the line of sight; user 10's eyes are either left eye 11 or right eye 12. For example, user 10's left eye 11 is focused on target 30 according to the direction of the line of sight LS1, and right eye 12 is focused on target 30 according to the direction of the line of sight LS2. The correction system 1000 corrects the line of sight LS1 of user 10's left eye 11 and the line of sight LS2 of user 10's right eye 12 separately. The embodiments described below are only illustrative examples of the correction of the line of sight LS1 of the left eye 11 (the technical solutions of each embodiment can also be applied to the correction of the line of sight LS2 of the right eye 12).

[0068] Figure 2This is a schematic diagram of the line of sight LS1 of the left eye 11 of user 10. The line of sight LS1 can be specifically described using a coordinate system defined by the X-axis, Y-axis, and Z-axis. The eye position e1 of the left eye 11 serves as the reference point of the coordinate system, through which the X-axis, Y-axis, and Z-axis all pass. The Z-axis is, for example, substantially perpendicular to the ground; when user 10 is seated inside the vehicle, the upper torso 13 of user 10 is substantially parallel to the Z-axis. On the other hand, the X-axis and Y-axis are both orthogonal to the Z-axis; the X-axis is, for example, the frontal direction of the upper torso 13 of user 10.

[0069] The line of sight LS1 of the left eye 11 extends from the eye position e1 to the target position p1 of the target object 30. The line of sight LS1 forms an angle A1 with the X-axis, and its direction can be defined based on this angle A1. In three-dimensional space, the line of sight LS1 also forms an angle with the Z-axis, and... Figure 2 The embodiment does not show the angle between the line of sight LS1 and the Z-axis. The direction of the line of sight LS1 is illustrated only by way of example in the two-dimensional plane formed by the X-axis and Y-axis. In the embodiments described below, the correction system 1000 corrects the line of sight LS1 by taking the line of sight LS1 of the user 10's left eye 11 as an example.

[0070] Figure 3 This is a block diagram of a correction system 1000 according to an embodiment of the present invention. Figure 4 A schematic diagram showing the calibration system 1000 installed on the vehicle 20. Please also refer to... Figure 3 and Figure 4 The correction system 1000 includes a reference result calculation device 100, a filtering processing device 200, a compensation value calculation device 300, a configuration management device 400, and a line-of-sight correction device 500. The reference result calculation device 100 includes an eye position calculation unit 101 and a spatial conversion unit 103.

[0071] In one example, the correction system 1000 is a hardware circuit located inside the vehicle, such as a hardware processor including, but not limited to, a digital signal processor (DSP), a central processing unit (CPU), and a micro control unit (MCU). The reference result calculation device 100, the filtering processing device 200, the compensation value calculation device 300, the configuration management device 400, and the line-of-sight correction device 500 are hardware circuit units within the correction system 1000.

[0072] In another example, the correction system 1000 is implemented as a software program module; the overall function of the correction system 1000 is achieved by executing the internal software program code (which includes multiple instructions) of a hardware processor (e.g., digital signal processor, central processing unit, or microcontroller unit) or hardware device (e.g., controller, computer device, or computer). Furthermore, the reference result calculation device 100, filtering processing device 200, compensation value calculation device 300, configuration management device 400, and line-of-sight correction device 500 within the correction system 1000 are all software modules, and their respective functions are implemented through software program code. The aforementioned software program code can be stored in a non-transitory computer-readable storage medium; such a non-transitory computer-readable storage medium is, for example, various forms of non-transitory (non-volatile) memory, hard disk, USB flash drive, etc. The non-transitory computer-readable storage medium can be electrically connected to the hardware processor or hardware device, or the non-transitory computer-readable storage medium is disposed within the hardware processor or hardware device. When a hardware processor or hardware device reads the aforementioned software program code from a non-transitory computer-readable storage medium, the hardware processor or hardware device can execute the instructions in the software program code to realize the respective functions of the reference result calculation device 100, the filtering processing device 200, the compensation value calculation device 300, the configuration management device 400, and the line-of-sight correction device 500.

[0073] The correction system 1000 operates in conjunction with the image acquisition device 21, the sensing system 22, and the gaze prediction device 40. The image acquisition device 21 is a camera installed inside the vehicle 20; for example, a rearview mirror 25 can be mounted on the inner side of the vehicle roof, and the image acquisition device 21 can be mounted on the rearview mirror 25, with its lens facing the user 10. The image acquisition device 21 acquires images of the user 10's face to generate a first image M1. The first image M1 includes the user 10's eyes, for example, the user's left eye 11. The image acquisition device 21 generates the first image M1 and transmits it to the eye position calculation unit 101 of the reference result calculation device 100.

[0074] The eye position calculation unit 101 performs image processing based on the first image M1 to calculate the eye position e1 of the left eye 11. Please also refer to... Figure 2 The eye position e1 of the left eye 11 can be further defined by the coordinates {e1} of the X-axis, Y-axis, and Z-axis. x e1 y e1 z}express.

[0075] Next, please also see Figure 4The vehicle 20 is further equipped with a sensing system 22. The sensing system 22 is also an external component independent of the calibration system 1000. The sensing system 22 includes a sensing device 221 and a target position calculation unit 222. The sensing device 221 is, for example, a radar sensing device or a lidar sensing device. The sensing device 221 senses (e.g., measures the distance) the target object 30 outside the vehicle to obtain a sensing result S1 of the target object 30. The sensing device 221 transmits the sensing result S1 associated with the target object 30 to the target position calculation unit 222. The target position calculation unit 222 analyzes the sensing result S1 to calculate the target position p1 of the target object 30. See also... Figure 2 The target position p1 of target object 30 can be further defined by the coordinates of the X-axis, Y-axis, and Z-axis {p1}. x p1 y p1 z}express.

[0076] Then, the spatial conversion unit 103 of the reference result calculation device 100 calculates the angle A1 between the line of sight LS1 of the left eye 11 and the X-axis based on the eye position e1 and the target position p1; and the spatial conversion unit 103 defines the direction of the line of sight LS1 according to the angle A1 between the line of sight LS1 and the X-axis. The reference result calculation device 100 uses the line of sight LS1 as the reference line of sight LS. REF And take the included angle A1 as the reference line-of-sight angle A REF .

[0077] As described above, the target position calculation unit 222 of the sensing system 22 analyzes the sensing result S1 obtained by the sensing device 221 to obtain the target position p1 of the target object 30. This position has high accuracy and therefore high reliability, reflecting the true position of the target object 30. In other words, the reference result calculation device 100 obtains the reference line of sight LS based on the target position p1. REF Angle A relative to the reference line of sight REF It has a high degree of reliability. (Reference: LS) REF Angle A relative to the reference line of sight REF This can be collectively referred to as the reference result REF; the reference result REF is more likely to reflect the true state of the user 10's line of sight LS1 (i.e., "Ground Truth, GT"). Furthermore, the reference result calculation device 100 transmits the reference result REF to the compensation value calculation device 300.

[0078] Image acquisition device 21 also transmits the first image M1, including the left eye 11, to gaze prediction device 40 for gaze prediction. Just as image acquisition device 21 and sensing system 22 are external devices separate from the correction system 1000 of the present invention, gaze prediction device 40 is also separate from the correction system 1000 of the present invention.Figure 3 In this example, the gaze prediction device 40 and the image acquisition device 21 are two separate devices. Alternatively, in other examples, the gaze prediction device 40 may be integrated into the image acquisition device 21.

[0079] In operation, based on the first image M1 generated by the image acquisition device 21, the gaze prediction device 40 performs gaze prediction on the gaze LS1 of the left eye 11 to generate a first gaze prediction result PD1 for the user 10's left eye 11. More specifically, the image acquisition device 21 performs multiple image acquisitions on the user 10's left eye 11 at multiple time points within a specific time interval to generate multiple first images M1 respectively; and the gaze prediction device 40 performs multiple gaze predictions on the aforementioned multiple first images M1 to generate multiple first gaze prediction results PD1 corresponding to multiple time points.

[0080] The line-of-sight prediction made by the line-of-sight prediction device 40 is only a preliminary prediction, and the stability of the resulting first line-of-sight prediction result PD1 may be low. Therefore, the line-of-sight prediction made by the line-of-sight prediction device 40 is called a "coarse prediction".

[0081] Please see Figure 5A This is a schematic diagram of the first gaze prediction result PD1. As mentioned above, the gaze prediction device 40 performs multiple gaze predictions based on multiple first images M1 at multiple time points to generate multiple first gaze prediction results PD1; and Figure 5A Only one of the multiple first prediction results PD1 is shown. The first gaze prediction result PD1 includes the predicted gaze direction and gaze angle of the left eye 11. Specifically, the first gaze prediction result PD1 includes the first predicted gaze LS. PD1 With the first predicted line of sight angle A PD1 As shown in equation (1):

[0082] PD1 = {LS PD1 A PD1} (1)

[0083] First Predictive Line of Sight (LS) PD1 This indicates the predicted gaze direction of the left eye 11, and is expressed as the first predicted gaze angle A. PD1 This represents the predicted viewing angle of the left eye 11. Wherein, the first predicted viewing angle A... PD1 For example, the first predictive line of sight (LS) PD1 The angle between the X-axis and the X-axis. In other words, the multiple first-line prediction results PD1 obtained by making multiple predictions at multiple time points may include multiple first-line prediction lines LS. PD1 With the corresponding multiple first predicted line-of-sight angles A PD1The line-of-sight prediction device 40 transmits multiple first line-of-sight prediction results PD1 to the filtering processing device 200.

[0084] The filtering processing device 200 filters multiple first line-of-sight prediction results PD1 to generate a second line-of-sight prediction result PD2. The filtering processing performed by the filtering processing device 200 includes, for example, statistical processing of the multiple first line-of-sight prediction results PD1, which calculates statistical parameters for the multiple first line-of-sight prediction results PD1. In this embodiment, the statistical processing includes, for example, calculating the probability distribution of the multiple first line-of-sight prediction results PD1, which can calculate multiple first predicted line-of-sight angles A. PD1 The average value E_A PD1 and these first predicted line-of-sight angles A PD1 The variation of the probability distribution V_A PD1 Furthermore, these first predicted line-of-sight angles A PD1 The distribution range R0 has a probability distribution. The center of the distribution range R0 corresponds to the mean E_A. PD1 Furthermore, the size of the probability interval of the distribution range R0 depends on the variance V_A. PD1 .

[0085] Next, please see Figure 5B This is a schematic diagram of the second line-of-sight prediction result PD2. The filtering processing device 200 uses multiple first predicted line-of-sight angles A PD1 The average value E_A PD1 As the second predicted line-of-sight angle A PD2 In other words, the filtering device 200 filters multiple first predicted viewing angles A. PD1 Take the average to obtain the second predicted line-of-sight angle A PD2 .

[0086] Furthermore, the second predicted viewing angle A PD2 The corresponding line of sight is the second predicted line of sight (LS). PD2 The filtering device 200 will filter the second predicted viewing angle A. PD2 With the second predictive line of sight LS PD2 This is integrated into the second line-of-sight prediction result PD2. That is, the second line-of-sight prediction result PD2 includes the second predicted line-of-sight angle A. PD2 With the corresponding second predictive line of sight LS PD2 As shown in equation (2):

[0087] PD2 = {LS PD2 A PD2} (2)

[0088] As mentioned earlier, the gaze prediction device 40 performs a coarse prediction to generate multiple first gaze prediction results PD1; while the second gaze prediction result PD2 is a further filtered version of the multiple first gaze prediction results PD1. Therefore, the second gaze prediction result PD2 has higher stability; and the filtering process performed by the filtering device 200 can be referred to as "fine prediction".

[0089] The filtering processing device 200 transmits the second line-of-sight prediction result PD2 to the compensation value calculation device 300. Meanwhile, the compensation value calculation device 300 receives the reference result REF generated by the spatial conversion unit 103 of the reference result calculation device 100. The compensation value calculation device 300 estimates the offset B based on the second line-of-sight prediction result PD2 and the reference result REF. The offset B serves as a compensation value, and its function is to compensate for the first line-of-sight prediction result PD1 generated by the line-of-sight prediction device 40 in the future.

[0090] Please see Figure 5C This is a schematic diagram of the offset B. Offset B is the reference line of sight LS in the reference result REF. REF The second predicted line of sight (LS) in the second line of sight prediction result PD2 PD2 The degree of offset between the two (i.e., the degree of error of the second line-of-sight prediction result PD2 compared to the reference result REF). In one example, the reference line-of-sight angle A is used. REF With the second predicted line of sight angle A PD2 The difference between them is used as the offset B.

[0091] Furthermore, the compensation value calculation device 300 is based on the reference line-of-sight angle A. REF Offset B and variation V_A PD1 Define the effective region R1 (as mentioned before, the variance V_A) PD1 It is multiple first-predicted line-of-sight angles A at multiple time points. PD1 (The variation of the probability distribution).

[0092] Next, please see Figure 6A and 6B This is a schematic diagram of the effective region R1. Figure 6A In this context, the effective region R1 is defined based on the reference center and the span C; the effective region R1 extends outward from the reference center with a span C. The reference center of the effective region R1 corresponds to the reference line of sight angle A. REF Reference line of sight LS REF Furthermore, the expansion C of the effective region R1 is, for example, equal to the variation V_A. PD1 The product of the multiplier f and the offset B is summed. The multiplier f is, for example, a positive integer or a decimal, such as a positive integer "3". In other words, with reference line of sight angle A... REFThe angle obtained by subtracting the span C is used as the "starting angle A_s", with reference to the line-of-sight angle A. REF Adding the angle obtained by the span C as the "termination angle A_e", the effective area R1 is the angle range covered by the "starting angle A_s" to the "termination angle A_e".

[0093] The span C is equal to the variation V_A PD1 The specific meaning of summing the product of the multiplier f and the offset B is: to increase the first predicted line-of-sight angle A. PD1 The variation V_A PD1 With the second predicted line of sight angle A PD2 The offset B is taken into consideration to evaluate the overall error of the line-of-sight prediction device 40; in other words, the span C can represent the overall error of the line-of-sight prediction device 40. Therefore, the effective region R1 expanded according to the span C can represent the possible angular range of the target position p1 of the target object 30. Each angle within the effective region R1 can be the angle of the target position p1. If more than two targets are detected within the effective region R1 (i.e., the effective region R1 includes other targets 32 besides the target object 30; the target position p2 of the target object 32 falls within the angular range of the effective region R1), then the second line-of-sight prediction result PD2 obtained by the line-of-sight prediction device 40 and the collaborative filtering processing device 200 in this prediction is considered invalid.

[0094] Conversely, if there is only a single target 30 within the effective area R1, then the second line-of-sight prediction result PD2 generated this time is considered valid, and its corresponding offset B (i.e., reference line-of-sight angle A) is considered valid. REF The second predicted line-of-sight angle A generated this time PD2 The difference between the two values ​​is considered valid. The compensation value calculation device 300 transmits the valid offset B to the configuration management device 400. Furthermore, the configuration management device 400 updates the configuration file with this valid offset B.

[0095] Figure 6A The coverage area of ​​the effective region R1 is illustrated using only a two-dimensional plane along the X-axis and Y-axis as an example. See also... Figure 6B The effective region R1 is described using a three-dimensional space with X-axis, Y-axis, and Z-axis. xz The scope of coverage. In terms of three-dimensional space along the X-axis, Y-axis, and Z-axis, the reference line of sight LS REF Pointing to the valid region R1 xz The reference center is (i.e., the effective region R1) xz The reference center is the target position p1 of target object 30. Furthermore, the effective area R1... xz The span has two dimensions, namely span C. x With span C z Among them, the exhibition width Cx System Figure 6A The spread C shown is the angular range of the effective region R1 in the two-dimensional plane along the X-axis and Y-axis. Furthermore, the spread C... z For the effective region R1 xz The extended angular range in the two-dimensional plane between the Y and Z axes. Spread C z The definition is similar to the span C x ; Spread width C z It is equivalent to the reference line of sight (LS). REF Angle with the Z-axis ( Figure 6B The sum of the offset and the multiple of the variation (not shown). In judging the validity of the second line-of-sight prediction result PD2, if the target position p2 of other target objects 32 besides target object 30 falls into the valid area R1... xz If the effective area R1 is within the specified range, then the second line-of-sight prediction result PD2 obtained by the line-of-sight prediction device 40 and the collaborative filtering processing device 200 in this prediction is determined to be invalid. Conversely, if the effective area R1 is within the specified range, then the second line-of-sight prediction result PD2 obtained by the collaborative filtering processing device 200 in this prediction is determined to be invalid. xz If only the target position p1 of target object 30 is detected, the second line-of-sight prediction result PD2 obtained in this prediction is judged as valid. The configuration management device 400 only updates the offset B corresponding to the second line-of-sight prediction result PD2 that is judged as valid in the configuration file.

[0096] In summary, based on Figure 3 , 5A In embodiments ~5C, 6A, and 6B, when the user 10's left eye 11 gazes at the target object 30 from a certain viewing angle, the image acquisition device 21 generates a first image M1 including the left eye 11, and the gaze prediction device 40 predicts the gaze LS1 of the left eye 11 based on the first image M1 to obtain a first gaze prediction result PD1. The correction system 1000 can decide whether to include the first gaze prediction result PD1 in the compensation calculation process based on preset conditions. Preset conditions may include, for example, the complexity of the external environment of the vehicle; if the complexity of the external environment is high, the correction system 1000 will not include the first gaze prediction result PD1 in the compensation calculation process. In one example, the correction system 1000 may refer to the sensing result S1 generated by the sensing device 221 to determine the complexity of the external environment of the vehicle. If the sensing result S1 indicates that the number of objects in the external environment of the vehicle is large (e.g., the number of objects exceeds an upper limit), the correction system 1000 will not include the first gaze prediction result PD1 in the compensation calculation process. If the sensing result S1 indicates that the number of objects in the environment outside the vehicle is less than the upper limit, then the first line-of-sight prediction result PD1 is included in the compensation calculation process.

[0097] Furthermore, the filtering processing device 200 of the correction system 1000 performs filtering processing based on the adopted first line-of-sight prediction result PD1 to obtain a second line-of-sight prediction result PD2 (including the second predicted line of sight LS).PD2 With the second predicted line of sight angle A PD2 On the other hand, the reference result calculation device 100 of the correction system 1000 calculates the reference result REF (including the reference line of sight LS) of the left eye 11 based on the first image M1 and the target position p1 of the target object 30. REF Angle A relative to the reference line of sight REF Then, the compensation value calculation device 300 of the correction system 1000 calculates the offset B based on the second line-of-sight prediction result PD2 and the reference result REF. Furthermore, the compensation value calculation device 300 can determine whether the second line-of-sight prediction result PD2 is valid based on the number of targets within the effective area R1; if the second line-of-sight prediction result PD2 is valid, the correction system 1000 can adopt the offset B corresponding to the valid second line-of-sight prediction result PD2, and the compensation value calculation device 300 transmits the adopted offset B to the configuration management device 400, which updates the configuration file with the adopted offset B. Similarly, the user 10's left eye 11 can gaze at other targets from another line-of-sight angle, and the compensation value calculation device 300 calculates the offset B corresponding to this line-of-sight angle. Furthermore, the compensation value calculation device 300 can calculate the offset B for each of the multiple viewing angles within a predetermined angle range, and update these offsets B in the configuration file of the configuration management device 400. The configuration management device 400 will continuously record the offset B corresponding to each viewing angle, as detailed below.

[0098] Please see Figure 7A This is a schematic diagram of the compensation value calculation device 300 calculating the offset B within a predetermined angle range and setting correction points. The compensation value calculation device 300 calculates the offset B corresponding to the second line-of-sight prediction result PD2 at different angles. For example, the offset B corresponding to the second line-of-sight prediction result PD2 is calculated at 20 degrees, 35 degrees, 50 degrees, 65 degrees, and 80 degrees respectively: the offset B calculated at 20 degrees is "3.3 degrees", the offset B calculated at 35 degrees is "3.5 degrees", the offset B calculated at 50 degrees is "3.1 degrees", the offset B calculated at 65 degrees is "2.8 degrees", and the offset B calculated at 80 degrees is "2.9 degrees". The offsets B of "3.3 degrees", "3.5 degrees", "3.1 degrees", "2.8 degrees", and "2.9 degrees" mentioned above are discretized in a binning manner and set as correction points.

[0099] Based on the above mechanism, the compensation value calculation device 300 sets multiple correction points, for example, five correction points, which correspond to 20 degrees, 35 degrees, 50 degrees, 65 degrees, and 80 degrees respectively. When the line-of-sight prediction device 40 predicts the first predicted line-of-sight angle A... PD1If the line of sight correction device 500 is equal to one of the five correction points of 20 degrees, 35 degrees, 50 degrees, 65 degrees, and 80 degrees, then the line of sight correction device 500 uses the first predicted line of sight angle A recorded in the configuration file. PD1 The average offset B of the correction points corresponding to equal angles (called the "average offset B'") is directly related to the first predicted line-of-sight angle A in the future. PD1 Compensation is performed. That is, the first predicted line-of-sight angle A is adjusted. PD1 The average offset B' of the correction points corresponding to equal angles plus the first predicted line-of-sight angle A PD1 The corrected line-of-sight prediction result PD1b is obtained.

[0100] If the first predicted line-of-sight angle A is obtained from the future prediction PD1 If the line of sight correction is not equal to any of the five correction points of 20 degrees, 35 degrees, 50 degrees, 65 degrees, and 80 degrees, then the line of sight correction device 500 uses the first predicted line of sight angle A. PD1 The average offset corresponding to the interval falls into is compensated. An interval is, for example, the range between two adjacent correction points. For instance: the first interval SEG1 is the range between two correction points at 20 degrees and 35 degrees; the second interval SEG2 is the range between two correction points at 35 degrees and 50 degrees; the third interval SEG3 is the range between two correction points at 50 degrees and 65 degrees; and the fourth interval SEG4 is the range between two correction points at 65 degrees and 80 degrees. In one example, the first predicted line-of-sight angle A falling within the same interval... PD1 All were compensated using the same average offset to obtain the corrected gaze prediction result PD1b. Please refer to [link / reference]. Figure 7B It is the first predicted line-of-sight angle A. PD1 A schematic diagram illustrating correction based on the average offset of the corresponding interval. First predicted line-of-sight angle A PD1 For example, 30 degrees falls within the first interval SEG1; any first predicted line-of-sight angle A falling within the first interval SEG1 PD1 All are compensated using the same average offset B'_1. The average offset B'_1 for the first interval SEG1 is calculated using the average offset B = 3.3 degrees, which is the 20-degree correction point at one endpoint of SEG1. The first predicted viewing angle A is... PD1 Adding the average offset B'_1 (3.3 degrees) results in the compensated viewing angle C_A (33.3 degrees). Following this method, the same average offset is used to predict the first viewing angle A at different angles within the same interval. PD1 Compensation can simplify computational complexity and reduce data volume.

[0101] Similarly, the average offset B' (3.5 degrees) of the correction point at one endpoint of the second interval SEG2 is taken as the offset B'_2 corresponding to the second interval SEG2; any first predicted line-of-sight angle A falling within the second interval SEG2 is... PD1 All are compensated with the same average offset B'_2 (the first predicted line of sight angle A). PD1 Add the average offset B'_2). Similarly, any first predicted line-of-sight angle A falling within the third interval SEG3... PD1 All are compensated with the same average offset B'_3, and any first predicted line-of-sight angle A falling into the fourth interval SEG4. PD1 All are compensated with the same average offset B'_4.

[0102] In another example, the gaze correction device 500 uses a first predicted gaze angle A PD1 The average offset B' of the two closest correction points is interpolated, and the interpolated value is used to compensate for the first predicted viewing angle A. PD1 Please see [link / reference] Figure 7A Taking linear interpolation as an example, if the first predicted viewing angle A PD1 If the angle is 25 degrees, then the compensation is performed by linear interpolating the average offset B' (3.3 degrees) of the nearest correction point of 20 degrees and the average offset B' (3.5 degrees) of the correction point of 35 degrees. The linear interpolation of the average offset B' (3.3 degrees) and the average offset B' (3.5 degrees) is equal to the sum of 3.3 degrees multiplied by 2 / 3 and 3.5 degrees multiplied by 1 / 3.

[0103] In yet another example, the gaze correction device 500 uses a first predicted gaze angle A PD1 The average offset B' of the correction point corresponding to the closest angle is used for compensation.

[0104] exist Figure 7A , 7B In this embodiment, the interval between the multiple correction points is, for example, 15 degrees. And, please see... Figure 7C This is another schematic diagram of the compensation value calculation device 300 setting the correction point; in Figure 7C In one embodiment, the interval between multiple correction points is, for example, 1 degree (the angle of the correction point is, for example, 20 degrees, 21 degrees, 22 degrees, etc.).

[0105] Next, please see Figure 8 , it is Figure 3 A detailed schematic diagram illustrating the operation of the calibration system 1000. Figure 8In the example, each signal has independent variables "1", "2", ..., "n" and "n+1", representing time points t1, t2, ..., t(n) and t(n+1), respectively. Each time point from t1 to t(n) can represent a "historical time point" of the operation of the correction system 1000, while time point t(n+1) represents a "future time point" of the operation of the correction system 1000.

[0106] Image acquisition device 21 generates multiple first images M1(1,2,…,n+1) at time points t1 to t(n+1). Furthermore, sensing device 221 of sensing system 22 generates multiple sensing results S1(1,2,…,n+1) at time points t1 to t(n+1). Target position calculation unit 222 of sensing system 22 generates multiple target positions p1(1,2,…,n+1) associated with target object 30 based on the sensing results S1(1,2,…,n+1).

[0107] At historical time points t1 to t(n), the reference result calculation device 100 of the correction system 1000 operates, wherein the eye position calculation unit 101 calculates multiple eye positions e(1,2,…,n) based on the first image M1(1,2,…,n) at historical time points t1 to t(n). Furthermore, the spatial conversion unit 103 calculates the reference result REF(1,2,…,n) based on the target position p1(1,2,…,n) at historical time points t1 to t(n).

[0108] On the other hand, the gaze prediction device 40 generates multiple first gaze prediction results PD1(1,2,…,n) based on the first images M1(1,2,…,n) from historical time points t1 to t(n). The filtering processing device 200 performs filtering processing based on the first gaze prediction results PD1(1,2,…,n) to generate multiple second gaze prediction results PD2(1,2,…,n).

[0109] At historical time points t1 to t(n), the compensation value calculation device 300 calculates multiple offsets B(A) based on the reference result REF(1,2,…,n) and the second line-of-sight prediction result PD2(1,2,…,n). PD2 Offset B(A) PD2 The associated second predicted line-of-sight angle A PD2 The angle corresponding to the correction point. The compensation value calculation device 300 can determine the offset B(A) PD2 Is it valid? If valid, then set the offset B(A) to 1.5. PD1 The data is transmitted to the configuration management device 400 and updated in the configuration file. In the configuration file, the offset B (A) is... PD2 The associated first predicted line-of-sight angle APD2 Establish as a calibration point.

[0110] At a future time point t(n+1), the gaze prediction device 40 generates a first gaze prediction result PD1(n+1) based on the first image M1(n+1). The first gaze prediction result PD1(n+1) includes a first predicted gaze angle A. PD1 (n+1) can be referred to as the "future predicted line-of-sight angle". Furthermore, the line-of-sight correction device 500 of the correction system 1000 operates at a future time point t(n+1), obtaining the second predicted line-of-sight angle A at the same angle as the first predicted line-of-sight result PD1(n+1) from the configuration file of the configuration management device 400. PD2 The average offset B'(A) of the corresponding correction point PD2 Furthermore, the gaze correction device 500 adjusts the gaze based on the average offset B'(A). PD2 The first predicted viewing angle A generated by the viewing prediction device 40 PD1 The (n+1) (i.e., the future predicted viewing angle) is compensated to produce the corrected viewing prediction result PD1b(n+1).

[0111] Figure 9A , 9B This is a flowchart of a correction method according to an embodiment of the present invention. The correction method of this embodiment, for example, is achieved by... Figure 3 The calibration system 1000 is implemented. Please refer to [link / reference needed]. Figure 9A First, step S900 is executed: the reference result calculation device 100 receives the first image M1 generated by the image acquisition device 21, which includes the user 10's eye (e.g., left eye 11), and receives the target position p1 generated by the sensing system 22.

[0112] Next, step S902 is executed: the first image M1 of the eye is analyzed by the eye position calculation unit 101 to obtain the eye position e1.

[0113] Next, step S904 is executed: the spatial transformation unit 103 calculates the reference line-of-sight angle A based on the eye position e1 and the target position p1. REF .

[0114] Next, step S906 is executed: the gaze prediction device 40 performs a coarse prediction based on the first image M1 to obtain multiple first gaze prediction results PD1, which include multiple first predicted gaze angles A. PD1 .

[0115] Next, step S908 is executed: the filtering processing device 200 filters the multiple first line-of-sight prediction results PD1 to obtain a second line-of-sight prediction result PD2. The second line-of-sight prediction result PD2 includes a second predicted line-of-sight angle A. PD2Filtering, for example, is performed on multiple first predicted viewing angles A. PD1 Perform statistical processing to calculate the first predicted line-of-sight angle A. PD1 The average value E_A PD1 With the variation V_A PD1 Furthermore, the average value E_A PD1 As the second predicted line-of-sight angle A PD2 .

[0116] Next, please see Figure 9B Step S910: The compensation value calculation device 300 calculates the compensation value based on the reference line-of-sight angle A. REF With the second predicted line of sight angle A PD2 Calculate the offset B.

[0117] Next, step S912 is executed: based on the reference line-of-sight angle A REF Offset B and variation V_A PD1 Define the valid region R1.

[0118] Next, step S914 is executed: determine whether there are any other targets besides target 30 in the effective area R1. If the determination result is "no" (that is, there is only a single target 30 in the effective area R1), then step S916 is executed: determine whether the current offset B is valid, and set the valid offset B as the correction point.

[0119] Then, step S918 is executed: the configuration management device 400 updates the offset B, which is valid and set as the correction point, in the configuration file.

[0120] Next, step S920 is executed: the line-of-sight correction device 500 adjusts the line of sight according to the first predicted line-of-sight angle A. PD1 The corresponding correction point is calculated by the average offset B', and the average offset B' is used to compensate for the first predicted viewing angle A currently predicted by the viewing prediction device 40. PD1 .

[0121] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A correction system, characterized by, The method comprises: calculating a reference gaze angle at each of a plurality of historical time points based on an eye position of an eye of a user and a target position associated with a target object, wherein the eye is a left eye or a right eye of the user, the user is located inside a vehicle, and the target object is located outside the vehicle; performing a filtering process on a plurality of first predicted gaze angles generated by a gaze prediction device at the historical time points to obtain a second predicted gaze angle at each of the historical time points, wherein the plurality of first predicted gaze angles are obtained by the gaze prediction device based on a plurality of first images associated with the eye; calculating a plurality of offsets based on the second predicted gaze angles and the reference gaze angles at the historical time points; updating the second predicted gaze angles and the offsets at the historical time points in a configuration file; calculating an average offset of the second predicted gaze angles based on the configuration file, and compensating a future predicted gaze angle based on the average offset. The plurality of first images associated with the eye are generated by an image acquisition device arranged inside the vehicle.

2. The correction system of claim 1, wherein, The method further comprises:

3. The correction system of claim 2, wherein, calculating the eye position by analyzing the plurality of first images associated with the eye; and calculating the reference gaze angle based on the eye position and the target position. The filtering process is performed to obtain a mean value and a variation of the plurality of first predicted gaze angles, and the second predicted gaze angle is equal to the mean value.

4. The correction system of claim 1, wherein, The method further comprises:

5. The correction system of claim 4, wherein, defining an effective region based on the reference gaze angle, the offset, and the variation, and determining whether the effective region contains other objects in addition to the target object.

6. The correction system of claim 5, wherein, The effective region has a reference center corresponding to the reference gaze angle and a spread equal to a product of the variation and a factor added to the offset.

7. The correction system of claim 5, wherein, When the effective region contains only the target object, the offset associated with the second predicted gaze angle is determined to be valid.

8. The correction system of claim 7, wherein, The method further comprises:

9. The correction system of claim 8, wherein, updating the valid offset in the configuration file.

10. The correction system of claim 9, wherein, The method further comprises:

11. A correction method characterized by, establishing a correction point based on the first predicted gaze angle associated with the offset. The method further comprises: compensating the future predicted gaze angle based on the average offset of the correction point corresponding to the future predicted gaze angle. The method comprises the following steps: generating, by a reference result calculation device, an eye position of an eye of a user at each of a plurality of historical time points, and obtaining a reference gaze angle at each of the historical time points according to the eye position and a target position associated with a target object, wherein the eye is a left eye or a right eye of the user, the user is located inside a vehicle, and the target object is located outside the vehicle; performing, by a filtering processing device, a filtering processing on a plurality of first predicted gaze angles generated by a gaze prediction device at the plurality of historical time points to obtain a second predicted gaze angle at each of the historical time points, wherein the first predicted gaze angles are obtained by the gaze prediction device according to a plurality of first images associated with the eye; obtaining, by a compensation value calculation device, a plurality of offsets according to the second predicted gaze angles and the reference gaze angles at the plurality of historical time points; updating, by a configuration management device, the second predicted gaze angles and the offsets at the plurality of historical time points in a configuration file; and calculating, by a gaze correction device, an average offset of the second predicted gaze angles according to the configuration file, and compensating a future predicted gaze angle according to the average offset.

12. The correction method of claim 11, wherein, wherein the first images associated with the eye are generated by an image acquisition device arranged inside the vehicle.

13. The correction method of claim 12, wherein, wherein the steps of generating the eye position and the target position by the reference result calculation device comprise: analyzing, by an eye position calculation unit of the reference result calculation device, the first images associated with the eye to obtain the eye position; and calculating, by a space conversion unit of the reference result calculation device, the reference gaze angle according to the eye position and the target position.

14. The correction method of claim 11, wherein, wherein the steps of performing the filtering processing by the filtering processing device comprise: calculating, by the filtering processing device, a mean value and a variation of the first predicted gaze angles; wherein the second predicted gaze angle is equal to the mean value.

15. The correction method of claim 14, wherein, comprising: defining, by the compensation value calculation device, an effective region according to the reference gaze angle, the offset, and the variation, and determining whether the effective region exists other target objects besides the target object.

16. The correction method of claim 15, wherein, wherein the effective region has a reference center and a spread, the reference center corresponds to the reference gaze angle, and the spread is equal to a product of the variation and a multiplier added to the offset.

17. The correction method of claim 15, wherein, wherein when the effective region only exists the target object, the second predicted gaze angle and the corresponding offset are determined to be valid.

18. The correction method of claim 17, wherein, comprising: updating, by the configuration management device, the valid offset in the configuration file.

19. The correction method of claim 18, wherein, comprising: establishing, by the compensation value calculation device, the first predicted gaze angle associated with the offset as a correction point.

20. The correction method of claim 19, wherein, comprising: compensating, by the gaze correction device, the future predicted gaze angle by the average offset of the correction point corresponding to the angle equal to the future predicted gaze angle.

21. A non-transitory computer-readable storage medium storing a plurality of instructions, the plurality of instructions comprising: when the instructions are read by a controller, a computer device, or a computer, the controller, the computer device, or the computer performs the correction method as claimed in any one of claims 11 to 20.

Images