Line-of-sight correction method, line-of-sight correction model training method, electronic equipment, program product and vehicle

Abstract

The invention provides a sight line correction method, a training method of a sight line correction model, electronic equipment, a program product and a vehicle, and relates to the technical field of images. In the application, the line-of-sight direction of the user can be determined based on the optical axis direction in the image through the line-of-sight correction model, so that the error caused by directly taking the optical axis direction as the line-of-sight direction is avoided, and the accuracy of line-of-sight direction detection is improved. In the training process of the sight line correction model, training data can be constructed based on the interactive operation of the user. Specifically, the sight line drop point when the user executes the interactive operation can be estimated based on the interactive operation, so that training data reflecting the difference between the sight line direction and the optical axis direction is constructed to train a sight line correction model. Therefore, the training data for model training can be obtained in response to the interactive operation at any time, and the data size and real-time performance of the training data are guaranteed. In addition, the construction of the training data does not need manual participation, and the non-sensibility of the correction process is realized.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a sight line correction method and a training method of a model thereof, an electronic device, a program product and a vehicle. Background Art

[0002] Gaze direction detection is an important technology in image processing technology. It mainly uses images including the user's eyes to estimate the user's gaze direction. Gaze direction detection technology has a wide range of applications. For example, in the field of vehicle technology, gaze direction detection technology can be used to determine whether the driver is paying attention during driving. For another example, in the field of interactive technology, especially in the fields of augmented reality and virtual reality, the display content can be adjusted according to the user's gaze direction to improve the user experience.

[0003] Therefore, how to accurately detect the direction of sight is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a line of sight correction method and a training method of its model, an electronic device, a program product and a vehicle, which can obtain the line of sight direction by correcting the user's optical axis direction, thereby achieving high-precision recognition of the line of sight direction.

[0005] In a first aspect, the present application provides a training method for a sight correction model, wherein the sight correction model is used to determine the sight direction of a target user based on the optical axis direction of the target user in an image. The training method includes: in response to an interactive operation of the target user on an interactive control, determining a user image of the target user when performing the interactive operation, wherein the user image includes an eye area of ​​the target user; determining the actual optical axis direction of the target user when performing the interactive operation based on the user image, wherein the actual optical axis direction passes through a visual positioning point of the eye area in the user image and an eye center point of the target user; determining an actual sight point based on the interactive operation, wherein the actual sight point includes a center position of the interactive control and / or a contact point of the target user on the interactive control; determining the actual sight direction of the target user when performing the interactive operation based on the actual sight point and the visual positioning point, wherein the actual sight direction points from the visual positioning point of the target user to the actual sight point; and training the sight correction model based on the actual sight direction of the target user in the user image and the actual optical axis direction.

[0006] In the second aspect, the present application provides a sight line correction method based on a sight line correction model. The method includes: collecting a user image of the target user, wherein the user image includes the eye area of ​​the target user; determining the current optical axis direction of the target user based on the visual positioning point in the user image, wherein the current optical axis direction passes through the visual positioning point and the eye center point of the target user; inputting the current optical axis direction into the sight line correction model, determining the current sight line direction through the sight line correction model, wherein the current sight line direction points from the visual positioning point of the target user to the actual sight line landing point, and the sight line correction model is trained based on the training method described in the first aspect.

[0007] In a third aspect, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the line of sight correction model described in the first aspect and / or the line of sight correction method based on the line of sight correction model described in the second aspect.

[0008] In a fourth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a computer program / instruction processor, implements the training method of the sight line correction model described in the first aspect and / or the sight line correction method based on the sight line correction model described in the second aspect. Optionally, the computer program can be stored in a readable storage medium or in the cloud of a computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0009] In a fifth aspect, the present application provides a vehicle, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor implements the training method of the line of sight correction model described in the first aspect and / or the line of sight correction method based on the line of sight correction model described in the second aspect.

[0010] The present application provides a line of sight correction method and a training method for its model. In the present application, the line of sight direction of the user can be determined by the line of sight correction model based on the direction of the optical axis in the image, thereby avoiding the error of directly taking the optical axis direction as the line of sight direction and improving the accuracy of line of sight direction detection. In the training process of the aforementioned line of sight correction model, training data can be constructed based on the user's interactive operation. Specifically, the line of sight landing point of the user when performing the interactive operation can be estimated based on the interactive operation, so as to construct training data that reflects the difference between the line of sight direction and the optical axis direction, so as to train the line of sight correction model. Thus, the training data for model training can be obtained at any time in response to the interactive operation, thereby ensuring the data volume and real-time nature of the training data. In addition, the construction of the training data does not require human participation, which realizes the non-sensitive correction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 It is a schematic diagram of an eyeball provided by an exemplary embodiment of the present application.

[0013] Figure 2 It is an exemplary flow chart of a line of sight direction detection method provided by an exemplary embodiment of the present application.

[0014] Figure 3 It is an exemplary flow chart of a method for determining an optical axis direction provided by an exemplary embodiment of the present application.

[0015] Figure 4 It is an exemplary flow chart of a sight line correction model configuration method provided by an exemplary embodiment of the present application.

[0016] Figure 5 This is a system module diagram of a sight line correction system provided in some embodiments of the present application.

[0017] Figure 6 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Application Overview:

[0020] In the related art, the recognition / detection of the gaze direction is generally performed based on the user's eye image, that is, the eye pupil position can be directly determined through the user's eye image to determine the user's gaze direction.

[0021] However, there is a deviation between the gaze detection result directly based on the pupil position and the user's actual gaze. That is, the pupil position generally reflects the direction of the optical axis, while the user's perceived gaze is generally the gaze direction, and there is a certain deviation between the two due to the physiological structure of the human body.

[0022] To further illustrate the difference between the visual axis and the optical axis of the human eye during observation, Figure 1 It is a schematic diagram of an eyeball provided by an exemplary embodiment of the present application.

[0023] like Figure 1 As shown, the eyeball 100 may include a corneal curvature center 110 reflecting the pupil position, an eyeball center 120 reflecting the eyeball geometric center, and a retinal fovea 130 .

[0024] The optical axis of the eyeball 100 may be a spatial axis passing through the corneal curvature center 110 and the eyeball center 120. The visual axis (ie, the line of sight direction) may be a spatial axis passing through the corneal curvature center 110 and the fovea 130 of the retina.

[0025] The optical axis can be the main path of light in the eyeball, that is, when light enters the eye from the outside, it is refracted by the cornea, then refracted by the lens, and finally focused on the retina. This propagation path is the optical axis, which can be replaced by the pupil center axis in practical applications. The visual axis refers to the straight line from the eye's gaze point through the retinal fovea 130 to the retina.

[0026] During actual observation, the visual axis and the optical axis often do not coincide with each other. The angle between the visual axis and the optical axis can be recorded as the Kappa angle.

[0027] Therefore, the sight direction determined in the above-mentioned related technologies is actually the optical axis direction, and there is a Kappa angle error with the actual sight direction. The Kappa angle itself is related to the physiological structure, and the Kappa angle errors of different people are different, and the actual visual axis and Kappa angle cannot be identified in the image.

[0028] To detect the direction of the line of sight, the relevant technology can first measure the user's Kappa angle through a series of active tests, and then correct the detected optical axis to determine the actual line of sight.

[0029] However, in the related art, considering that there are no obvious external features between the Kappa angle and the actual visual axis, the measurement process is often based on specific instructions and the user's response to the instructions. That is, the relevant staff can give the object to be measured an instruction to look at a certain object / direction, so that the object to be measured actually executes the instruction, thereby inferring the Kappa angle based on the known line of sight direction.

[0030] The above process has at least the following technical problems:

[0031] First, the entire data process is highly dependent on the interaction between the two parties, and users must match the corresponding instructions. The user experience of the entire process is poor and requires a lot of manpower costs.

[0032] Second, considering that the aforementioned measurement process requires annotation of specific objects and cooperation from both parties, the amount of data obtained is relatively small, but when there is abnormal data / the degree of cooperation is not high, it will seriously affect the accuracy of the results.

[0033] Therefore, in order to solve the aforementioned technical problems, the present application provides a line of sight correction method and a training method for its model. In the present application, the line of sight direction of the user can be determined by the line of sight correction model based on the optical axis direction in the image, thereby avoiding the error of directly taking the optical axis direction as the line of sight direction and improving the accuracy of line of sight direction detection. In the training process of the aforementioned line of sight correction model, training data can be constructed based on the user's interactive operations. Specifically, the line of sight landing point when the user performs the interactive operation can be estimated based on the interactive operation, so as to construct training data that reflects the difference between the line of sight direction and the optical axis direction to train the line of sight correction model. Thus, the training data for model training can be obtained at any time in response to the interactive operation, thereby ensuring the data volume and real-time nature of the training data. In addition, the construction of the training data does not require human participation, which realizes the non-sensitive correction process.

[0034] Various non-limiting embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0035] Exemplary gaze direction detection method:

[0036] Figure 2 It is an exemplary flow chart of a line of sight direction detection method provided by an exemplary embodiment of the present application.

[0037] In some embodiments, Figure 2The method P200 shown can be executed by a processing device. For example, P200 can be executed by a domain controller or other processing device in a vehicle. That is, the vehicle can detect the line of sight direction of people in the vehicle (such as drivers / passengers, etc.) based on the method shown in P200. For another example, P200 can be executed by a mobile terminal (such as a mobile phone, an augmented reality device, etc.), and the line of sight direction of the user using the terminal is identified through the sensor on the terminal.

[0038] like Figure 2 As shown, the aforementioned P200 may include the following steps:

[0039] S210: Collect a user image of a target user.

[0040] S220: Determine the current optical axis direction of the target user based on the visual positioning point in the user image.

[0041] S230, inputting the current optical axis direction into a sight line correction model, and determining the current sight line direction through the sight line correction model.

[0042] The target user may refer to the object for which the sight line correction is performed in this application. The target user may be configured according to the actual application scenario of the aforementioned P200. For example, the target user may be the user of the mobile terminal. For another example, the target user may be the driver of a vehicle.

[0043] Considering the availability of the gaze detection process, the gaze detection is generally performed based on image sensing data, that is, generally based on an image containing the target user to identify the gaze direction of the target user. Among them, the image containing the eye area of ​​the target user can be recorded as the user image. Among them, the user image can at least include the eye area of ​​the user.

[0044] In some embodiments, considering the complexity of spatial relationships, the aforementioned user image may be an image in a fixed scene. For example, it may be an in-car surveillance image containing the target user captured by an in-car camera. Thus, the scene where the user image is located is fixed, and spatial relationship mapping can be performed directly based on the user image, thereby estimating the three-dimensional spatial position of the target user's eye area in the fixed scene (such as the spatial coordinates in the fixed scene) based on the user's position in the two-dimensional image.

[0045] Correspondingly, for images in fixed scenes, the user image acquisition method is often fixed, thereby solidifying the spatial mapping relationship between the user image and the scene. Among them, the aforementioned fixation does not limit all parameters of the acquisition device. For example, changes in data such as lens orientation and focal length can be converted into internal and external parameters of the acquisition device (such as internal parameter matrix and external parameter matrix) to adjust the spatial relationship, and the mapping relationship can still be kept accurate.

[0046] As an exemplary illustration only, the aforementioned user image may be an image obtained by an in-vehicle camera device that includes at least the face of the target user. Then the scene corresponding to the user image may be the corresponding cockpit, that is, the three-dimensional coordinates of the corresponding position in the vehicle space may be estimated based on the position of the user (such as the eye area) in the user image, and then the line of sight of the target user in the in-vehicle space may be determined. Among them, the aforementioned user image may be an image of the entire target user captured by the sensor, or it may be a local image of the target user's facial area.

[0047] In some embodiments, the aforementioned user image may also be a three-dimensional image containing spatial information. Thus, the three-dimensional spatial position of the target user in the user image (such as the spatial coordinates under the three-dimensional image sensor coordinates) can be determined directly based on the spatial information of the three-dimensional image, and the line of sight direction of the target user in the three-dimensional image sensor spatial coordinate system can be determined.

[0048] In some embodiments, the aforementioned S210 may be performed according to actual sight line detection needs. For example, for driver distraction detection, user images may be periodically acquired to detect the direction of the driver's sight line, so as to avoid the driver's long-term distraction (such as lack of concentration due to fatigue driving) and reduce the possibility of accidents.

[0049] In the aforementioned S220, the visual positioning point may refer to a positioning point that can be determined from the image and reflects the target user's eye. Among them, considering the hiddenness of the eye in the physiological structure, the visual positioning point is the explicit feature of the target user's eye in the image. That is, the explicit feature of the eye may refer to the eye feature that can be identified from the image, generally the pupil. Then the visual positioning point may be a positioning point related to the pupil. For example, the visual positioning point may be the aforementioned center of corneal curvature, the center point of the pupil (such as the center point of the pupil at the lens, the corresponding position of the center point of the pupil on the outer surface of the eyeball, etc.). The current optical axis direction may refer to the direction of the optical axis when performing line of sight correction.

[0050] In some embodiments, the visual positioning point can be resolved into the position information of the structure in space, thereby reflecting the eye position of the target user. Among them, the position information contained in the visual positioning point can be represented as the position information in the application scene coordinate system (such as the in-vehicle coordinate system). For example, when the user image is acquired based on multiple cameras in the car, the visual positioning point can be represented as the position in the in-vehicle coordinate system (rather than the coordinate system under a specific sensor) to facilitate subsequent processing.

[0051] In some embodiments, considering the complexity of the aforementioned visual positioning point and the optical axis direction in space, the aforementioned S220 can be implemented by a machine learning model. That is, the user image containing the visual positioning point can be input into the optical axis determination model, so as to determine the optical axis direction of the target user in the user image (that is, the aforementioned current optical axis direction) by the optical axis determination model. Among them, the optical axis determination model can be a trained machine learning model.

[0052] In some embodiments, the aforementioned optical axis determination model can be trained directly based on the aforementioned fixed scene, thereby directly outputting an analytical expression of the optical axis direction in the fixed scene coordinate system.

[0053] In some embodiments, the visual positioning point can be marked in the user image when the optical axis determination model is input, so that the spatial analytical expression of the optical axis direction is determined based on the two-dimensional coordinates of the visual positioning point in the user image through the optical axis determination model. In some embodiments, the visual positioning point can also be unmarked when the optical axis determination model is input, so that the visual positioning point is identified from the user image through the optical axis determination mode, thereby determining the spatial analytical expression of the optical axis direction.

[0054] In some embodiments, it is considered that the direction of the optical axis can refer to the propagation direction of light in the eyeball (and its opposite direction). The direction of the optical axis can also be determined based on the structural characteristics of the eyeball, and the aforementioned S220 can also be implemented based on a spatial algorithm. Specifically, for example, the visual positioning point can be determined first, and the three-dimensional coordinates of the visual positioning point can be determined through the spatial mapping relationship between the user image and the scene. Then, based on the three-dimensional coordinates of the visual positioning point and the user image, an eyeball space model that matches the target user is constructed in the scene space. Finally, based on another point on the optical axis in the eyeball space model (which can be recorded as the center point of the eye, which can be the center point of the lens, the center point of the eyeball, etc.), the optical axis (that is, the spatial straight line passing through these two points) is determined through at least two positioning points (that is, the aforementioned visual positioning point and the center point of the eye).

[0055] Considering the large amount of computation required by the aforementioned spatial algorithm, the aforementioned process is generally used to construct training data for the optical axis determination model. The aforementioned optical axis determination model can be trained together with the aforementioned line of sight correction model. For specific training logic, please refer to Figure 4 And the related descriptions are not repeated here.

[0056] In the aforementioned S230, the sight correction model may refer to a related algorithm for correcting the direction of the optical axis, that is, the sight correction model may determine the current sight direction based on the current optical axis direction. Among them, the current sight direction (also recorded as the current visual axis direction) may reflect the sight direction determined during sight correction. The input of the sight correction model may be the current optical axis direction (such as the analytical expression of the optical axis in space), and the output may be directly the current sight direction, or it may be presented as the difference between the current sight direction and the current optical axis direction (such as the Kappa angle).

[0057] In some embodiments, the sight line correction model can be selected according to actual needs and calculation accuracy. For example, considering that the relationship between the optical axis direction and the sight line direction is generally approximately characterized as a linear relationship, the sight line correction model can select a linear regression algorithm. For another example, in order to further improve the calculation accuracy, the aforementioned sight line correction model can be configured as a machine learning algorithm (such as a deep learning model).

[0058] It should be noted that, considering that the line of sight direction generally points from the target user's visual positioning point to the actual line of sight landing point, the aforementioned line of sight correction model cannot be directly configured, and it is often necessary to train the model based on training data (i.e., perform parameter calculation / parameter iteration based on training data).

[0059] It is further considered that the line of sight direction is difficult to identify directly. The present application creatively constructs the training data of the aforementioned visual correction model based on interactive operations. That is, the algorithm parameters can be configured based on the actual optical axis direction and the actual line of sight landing point of the target user when performing interactive operations. Among them, the actual optical axis direction and the actual line of sight landing point of the target user when performing interactive operations can be used as training data to determine the relevant parameters of the algorithm. Among them, the actual line of sight landing point (or the processing result based on the actual line of sight landing point, such as the actual line of sight direction or the Kappa angle at the line of sight landing point) can be used as a sample label of the model, so as to determine the parameters of the machine learning model through model training.

[0060] When the target user performs an interactive operation, the target user's line of sight generally falls on the interactive control corresponding to the interactive operation. Based on this, the present application can construct the training data of the line of sight correction model based on the optical axis direction determined at this time (recorded as the actual optical axis direction) and the actual line of sight landing point (such as constructing the actual line of sight direction based on the actual line of sight landing point), thereby configuring the line of sight correction model so that the line of sight correction model can reflect the relationship between the optical axis direction and the line of sight direction of the target user (such as the Kappa angle), so that the line of sight direction can be determined based on the optical axis direction.

[0061] Interaction refers to the action or behavior performed when a person interacts with a computer, device, or other person. Specifically, when the present application is applied to a vehicle, an interaction may refer to an interaction triggered by a target user with a limb (such as a hand). For example, an interaction may include an interaction by the target user on a virtual interaction control on a vehicle screen and an interaction on an actual interaction control (such as a physical button, a gear lever, etc.).

[0062] The actual sight point of the interactive operation is generally the trigger position of the interactive operation. For example, the actual sight point can be the position of the interactive control corresponding to the interactive operation in space.

[0063] In some embodiments, considering that different users have different Kappa angles, the aforementioned sight line correction model can be configured based on different users, that is, the sight line correction model corresponds to the user one by one. For example, different sight line correction models are bound to the identity information of the corresponding user, and the sight line correction model corresponding to the user can be trained only based on the data of the user.

[0064] Taking the sight line correction model that outputs the Kappa angle as an example, when executing the aforementioned S230, the sight line correction model is called based on the identity information of the target user. Then the current optical axis direction is input into the sight line correction model, and the Kappa angle of the target user is determined by the sight line correction model. The current optical axis direction is corrected based on the Kappa angle to determine the current sight line direction. Among them, the sight line correction model is trained based on the data of the target user.

[0065] Therefore, the recognition of the line of sight direction in the aforementioned P200 is based on the correction of the user's optical axis direction. This process takes into account the line of sight deviation caused by the user's own body structure, thereby avoiding the error of directly using the optical axis direction as the line of sight direction, and improving the accuracy of line of sight direction recognition. In addition, the above correction process can be implemented based on the user's interactive operation, so that the line of sight correction process does not require the user's active cooperation, and the correction process is made unconscious. In particular, the line of sight correction provided by the present application is performed based on interactive operations, and the data for correction can be obtained at any time in response to interactive operations. The amount of data for correction is large and real-time, which further improves the accuracy of line of sight direction recognition.

[0066] In some embodiments, the aforementioned P200 can also be used as a processing method related to the viewing direction, that is, the aforementioned determined sight line direction can be further processed through subsequent steps. For example, the aforementioned P200 can be used as a driver sight line detection, then the aforementioned P200 can be executed by the processing device of the vehicle, the aforementioned target user can be the driver of the vehicle, and the aforementioned P200 can also detect the target user's distraction based on the sight line direction after the aforementioned S230.

[0067] In some embodiments, the aforementioned distraction detection can be implemented through a driver monitoring algorithm (Driver Monitor System, DMS), thereby determining whether the line of sight / line of sight direction meets normal driving requirements (such as looking directly at the road area ahead), and then detecting and reminding the driver of fatigue status / distraction status.

[0068] Specifically, in order to ensure the safety of the vehicle, the driver's line of sight should be directed to an effective area that can reflect the road conditions during driving. For example, the effective area may include the front of the vehicle, the rearview mirror / electronic rearview mirror, and the instrument panel. When performing distraction detection, it can be determined whether the driver's line of sight is directed toward the effective area based on the line of sight direction determined in the aforementioned steps, and based on the detection results (such as the length of time that the line of sight is not directed to the effective area, recorded as the distraction duration), it can be determined whether the driver is distracted (such as the aforementioned distraction duration is greater than the time threshold), so as to make appropriate reminders.

[0069] It should be noted that the detection conditions of distraction detection can be adjusted according to the actual application scenario to match the corresponding detection requirements. For example, if the vehicle is in high-level driving (such as when the vehicle is in L3 level or above autonomous driving mode), the relevant requirements for distraction detection may be lower (such as a larger time threshold for distraction duration).

[0070] In some embodiments, the aforementioned P200 can also be combined with the intelligent cockpit system when applied to vehicles, and after determining the sight point, interactive information can be provided based on the sight point. For example, based on the sight point of the driver (i.e., the target user), route information, driving status information mapping and other data can be displayed on the front glass through the vehicle head-up display technology (Head-up-Display, HUD), so as to provide prompt information accurately and efficiently.

[0071] Example sight correction model training process:

[0072] As mentioned above, the sight line correction model provided in the present application can be configured in response to the interactive operation of the target user, so as to achieve imperceptible sight line correction. That is, the training data of the sight line correction model can be determined based on the interactive operation of the target user, and the sight line correction model can be configured based on the training data.

[0073] Considering the differences in body structures of different people, the aforementioned sight line correction model corresponds to the target user one by one. That is, different target users need to be configured with different sight line correction models. The training data of the sight line correction model generally includes training samples and training labels, wherein the training samples generally include the actual optical axis direction of the target user when performing interactive operations, and the training labels in the training data generally include the actual sight line direction of the target user when performing interactive operations.

[0074] In some embodiments, considering that the type of the line of sight correction model can be adjusted according to the actual accuracy requirements, its configuration process can be performed based on its actual type. Among them, the configuration process of the line of sight correction model can be understood as a parameter adjustment process based on the aforementioned training data. For example, during the configuration process, the line of sight correction model can first determine the estimated line of sight direction based on the actual optical axis direction, and then update the parameters of the line of sight correction model based on the difference between the actual line of sight direction and the estimated line of sight direction. For another example, the correlation between the optical axis direction and the line of sight direction (or Kappa angle) can be directly fitted based on the training data, thereby configuring the model parameters of the line of sight correction model based on the correlation.

[0075] It should be noted that, considering that the line of sight correction model in the present application can be updated seamlessly based on the interactive operations of the target user, its initialization is generally completed during the target user's initial configuration process of the device (such as the initialization of the line of sight correction model during the driver's setting operation after picking up the car), and the line of sight correction model is continuously updated during the interactive process during subsequent actual use.

[0076] To further illustrate the process, the present application also provides an exemplary flow chart of the sight line correction model configuration process ( Figure 4 ).

[0077] like Figure 3 As shown, process P300 may include the following steps:

[0078] S310: In response to an interactive operation of a target user on an interactive control, determine a user image of the target user when performing the interactive operation.

[0079] S320: Determine, based on the user image, the actual optical axis direction of the target user when performing the interactive operation.

[0080] S330: Determine the actual sight point based on the interactive operation.

[0081] S340: Determine the actual sight direction of the target user when performing the interactive operation based on the actual sight point and the visual positioning point.

[0082] S350: training a sight line correction model based on the actual sight line direction and the actual optical axis direction of the target user in the user image.

[0083] In some embodiments, the aforementioned S310 may be executed in response to an interactive operation, that is, when the target user performs an interactive operation, the interactive operation generally generates a control signal for the device (such as a gear control of the vehicle, a switch of each function, etc.). The device may call a related camera device to obtain a corresponding user image when it learns the control signal. In addition, considering that the duration of the interactive operation is relatively long, the user image obtained may be a sequence of user images within the time period of performing the interactive operation.

[0084] In some embodiments, the present application may be executed based on the actual trigger of the interactive operation when executing S310. For example, when the actual trigger of the interactive operation is not an object that requires line of sight correction, S310 and its subsequent steps may not be executed. When the actual trigger is an object that requires line of sight correction, its user image is obtained to execute subsequent steps. Among them, the identification of the target user can be performed based on parameters such as its position in space and image features. The actual trigger of the interactive operation can be determined based on the interactive behavior of each candidate object in the user image sequence.

[0085] The actual optical axis direction may be the optical axis direction when the target user performs an interactive operation. In some embodiments, when executing the aforementioned S320, a target image matching the interactive operation (e.g., the eye orientation changes and is approximately toward the interactive control) may be first determined from the user image sequence, thereby determining a visual positioning point in the image based on the target image, and then determining the actual optical axis direction based on the visual positioning point.

[0086] In some embodiments, the aforementioned S320 may be based on Figure 2 As a preferred embodiment, the aforementioned S320 can be trained jointly with the optical axis determination model. That is, the training data of the optical axis determination model can be first constructed based on the spatial algorithm, and after the optical axis determination model is trained, the actual optical axis direction can be determined directly based on the optical axis determination model. For details, please refer to Figure 4 and its related description.

[0087] The actual sight point of an interactive operation may refer to the sight point of the target user when performing the interactive operation. It can generally be considered that the trigger position of the target user when performing the interactive operation can be used as the actual sight point of the interactive operation. Among them, the trigger position of the interactive operation may refer to the interaction position between the target user and the interactive control when performing the interactive operation. In this application, the trigger position can be used as the sight point of the target user when performing the interactive operation. That is, the target user first observes the corresponding interactive control, and then performs the corresponding interactive operation.

[0088] In some embodiments, the actual sight point can be determined based on the interactive control for interaction. That is, the interactive control for the interactive operation can be determined first. Thereby, the center position of the interactive control is used as the actual sight point or the contact point of the target user on the interactive control is used as the actual sight point. The specific determination method can be determined based on the type and perception ability of the interactive control. For example, when the interactive control is a physical button (such as a gear shift device, a horn, etc.), its center position can generally be selected as the actual sight point. When the interactive control is a button on the display screen, the contact point of the interactive control can generally be determined based on the display screen, thereby using it as the actual sight point.

[0089] In some embodiments, after determining the actual sight point, the actual sight direction (i.e., the sight direction of the target user when performing the interactive operation) can be parsed out. The actual sight direction points from the visual positioning point of the target user to the actual sight point, and is parsed as a function passing through two coordinates in space.

[0090] In some embodiments, the aforementioned S350 may configure the sight line correction model based on the training data determined in the aforementioned step. In addition, considering that there may be certain deviations based on the interactive operation itself, an algorithm / model with a certain tolerance for deviations may be selected when selecting the sight line correction model. For example, the sight line correction model may preferably be a deep learning model.

[0091] In some embodiments, in order to improve the accuracy of the sight line correction model and take into account the real-time nature of the configuration process, the confidence of the training data is first determined when executing the aforementioned S450. And the parameters of the sight line correction model are updated based on the training data whose confidence meets the threshold condition. That is, the currently acquired training data can be screened based on the confidence, so that the parameters of the sight line correction model are updated only based on the training data whose confidence is higher than the threshold.

[0092] In some embodiments, the confidence level may be determined based on the user's actual gaze. That is, in some cases, when the driver is familiar with certain interactive controls, the driver may operate the interactive controls (especially physical interactive controls) only by relying on memory and non-visual perception. For example, if the driver is familiar with the physical button positions, the driver may manipulate the relevant buttons or other interactive devices (such as a gear lever) without looking at the relevant positions to achieve the corresponding operations.

[0093] To avoid the above situation, when determining the confidence, it can be determined based on the eye changes whether the target user projects the interactive control of the interactive action when performing the interactive action. That is, the estimated sight direction can be determined based on the actual optical axis direction. Then, the confidence of the interactive operation is determined based on the difference between the estimated sight direction and the actual sight direction. Finally, in response to the confidence of the interactive operation being greater than the confidence threshold, the sight correction model is trained based on the actual sight direction of the target user in the user image and the actual optical axis direction.

[0094] The aforementioned estimated sight direction may be the correction result of the current optical axis direction based on the Kappa angle in this case. When determining the confidence level, the probability of the estimated sight direction and the actual sight direction coinciding may be determined based on the difference between the estimated sight direction and the actual sight direction (such as the difference in angle, the difference in landing point, etc.) as the aforementioned confidence level. That is, the higher the probability of the two coinciding, the more it indicates that the user has indeed cast a gaze on the interactive control at that moment.

[0095] In addition, considering that target users have different familiarity levels with different types of interactive controls, the aforementioned gaze probability may be adjusted in combination with the type of interactive control.

[0096] In some embodiments, in order to improve the accuracy of the actual line of sight correction model, in the training process of the aforementioned S350, multiple possible models can be trained, so as to select the model with the best training effect as the line of sight correction model. That is, multiple candidate correction models can be first trained based on the actual line of sight direction and the actual optical axis direction of the target user in the user image. In response to the completion of the training of any candidate correction model, the accuracy of the candidate correction model is determined. The line of sight correction model is determined from the multiple candidate correction models based on the accuracy of each candidate correction model.

[0097] Among them, the aforementioned accuracy can reflect the difference between the current sight direction output by the model and the actual sight direction. The accuracy can generally be performed after the model meets the training termination conditions (such as parameter convergence, the number of iterations meets the threshold conditions, etc.). The accuracy can generally be calculated based on a single acquisition of the user image. The difference between the aforementioned current sight direction and the actual sight direction determines the single validity (such as a difference of less than 2° is considered valid, otherwise it is invalid), thereby performing statistics on a large number of test structures to determine the accuracy.

[0098] The candidate correction model can be any trainable model / algorithm that can reflect the relationship between the sight line direction and the optical axis direction, so that the optimal algorithm can be selected and the reliability can be verified among the various algorithms. In actual work, considering the amount of calculation and the speed of calculation, the sight line correction model generally adopts a linear fitting algorithm.

[0099] Exemplary optical axis direction determination method:

[0100] To further illustrate the process of determining the optical axis direction, the present application also provides an exemplary flow chart of a method for determining the optical axis direction ( Figure 4 ).

[0101] As mentioned above, before the optical axis determination model is trained, it can be based on Figure 4 The optical axis is determined by the process in . When the optical axis determination model is trained, the visual positioning point can be input into the optical axis determination model, and the actual optical axis direction of the target user can be determined by the optical axis determination model. The optical axis determination model can determine the optical axis direction of the user in the user image.

[0102] like Figure 4 As shown, process P400 may include the following steps:

[0103] S410: Identify the visual positioning point of the target user from the user image.

[0104] S420: Determine the eye space model of the target user based on the visual positioning point.

[0105] S430: Determine the eye center point of the target user from the eye space model.

[0106] S440: Determine the actual optical axis direction based on the eye center point and the visual positioning point.

[0107] S450 , training an optical axis determination model based on the user image of the target user and the actual optical axis direction of the target user in the user image, until the optical axis determination model has completed training.

[0108] In some embodiments, the aforementioned S410 can be determined based on an image feature recognition algorithm. For example, the visual positioning point can be used as a target image feature, and the image feature can be identified from the user image through an image feature recognition algorithm, and then the coordinate data of the visual positioning point in the image is determined based on the recognition result (target selection box). Finally, a spatial transformation is performed to determine the coordinate data in the corresponding space.

[0109] The eyeball space model can be a three-dimensional eyeball model based on the physiological structure of the human body, which can reflect the structure of the human eyeball in space. For example, the eyeball space model can be constructed based on the conventional eyeball structure of the human body to reflect the eyeball structure of a normal human body.

[0110] In some embodiments, the aforementioned S420 can be constructed based on a structure in the eye space model that matches the visual positioning point, that is, the corresponding structure (such as the center point of the corneal curvature in the eye space model) can be spatially aligned with the visual positioning point, thereby constructing the eye space model in space (such as the aforementioned in-vehicle coordinate system).

[0111] In some embodiments, considering the differences in eye space models of different users, the construction process of S420 can also be adjusted based on the parameters of the target user. That is, S420 can further include the following steps:

[0112] S421. Determine the model position of the eyeball space model based on the position information of the visual positioning point.

[0113] S422: Determine the model size of the eyeball space model based on the size of the eye area in the user image.

[0114] S423: construct an eyeball space model based on the model position and model size.

[0115] In some embodiments, S421 may reflect the configuration of the eyeball space model position information. In practical applications, considering the meaning of the visual positioning point itself, its spatial information may include not only its coordinates in space, but also its orientation, wherein the orientation of the visual positioning point may reflect the orientation of the eyeball itself. For example, when the visual positioning point is the center point of corneal curvature, its orientation may be the direction of the line connecting the center point of corneal curvature and the pupil / the direction perpendicular to the tangent plane at the center position of corneal curvature, which may be determined specifically by the posture of the target user and / or the image of the user's eye in different user images, thereby representing the visual positioning point as a vector in space, the position of the vector being the position of the visual positioning point, and the direction of the vector being the orientation of the visual positioning point.

[0116] Based on the position information of the aforementioned visual positioning point, the structure corresponding to the eye space model can be aligned with the position information to determine the model position. The model position includes the spatial position of the eye space model and the eye direction. The spatial position of the model position can be determined by aligning the aforementioned structure, and the eye direction can be determined based on the direction of the aforementioned visual positioning point (generally the same).

[0117] In some embodiments, in order to make the eye space model fit the target user, the model size of the eye space model can be estimated based on the size of the eye area in the user image. The structure of the eye space model is generally similar, and its model size can generally be determined by scaling the model.

[0118] Based on the aforementioned model position and model size, the eye space model can be mapped in space to construct the eye space model of the target user.

[0119] It should be noted that the aforementioned eye space model may not be represented as a visible three-dimensional model, that is, in the aforementioned steps, a three-dimensional model may not be directly presented in the aforementioned space in a visual manner, but may be presented in the space in a spatial analytical manner. For example, the three-dimensional model parameters (such as feature points, feature functions and other data) of the eye space model may be mapped into the space, thereby realizing the construction of the eye space model.

[0120] Based on the eyeball space model determined above, the eyeball feature points can be extracted from the eyeball space model as the eyeball center point. Different from the aforementioned visual positioning point, the eyeball center point is generally an invisible structure inside the eyeball, and is generally configured as the geometric center of the eyeball structure (i.e., the aforementioned eyeball center 120). In addition, it should be noted that due to the differences in each body structure, the eyeball space model based on the average person cannot characterize the fovea 130.

[0121] After determining the aforementioned eye center point, based on Figure 1In the optical structure shown, the optical axis of the eyeball can be determined based on the aforementioned eyeball center point and the visual positioning point. Specifically, the optical axis direction passes through the eyeball center point and the visual positioning point at the same time, and the straight line function in space can be analyzed by the coordinates of the two points to serve as the optical axis direction.

[0122] Exemplary Systems and Electronic Devices:

[0123] Combination of the above Figures 1 to 4 , describes the method embodiment of the present application in detail, and the device embodiment of the present application is described in detail below. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the part not described in detail can refer to the previous method embodiment.

[0124] The present application also provides a sight line correction system, including a module for implementing the sight line direction detection method provided in the present application. Figure 5 As shown, Figure 5 It is a system module diagram of a line of sight correction system provided by an exemplary embodiment of the present application. The line of sight correction system 500 may include an image acquisition module 510, an optical axis recognition module 520 and a line of sight recognition module 530.

[0125] The image acquisition module 510 may be used to collect a user image of a target user, wherein the user image includes an eye area of ​​the target user.

[0126] The optical axis recognition module 520 may be used to determine the current optical axis direction of the target user based on the visual positioning point in the user image, wherein the current optical axis direction passes through the visual positioning point and the eye center point of the target user.

[0127] The line of sight recognition module 530 can be used to input the current optical axis direction into a line of sight correction model, and determine the current line of sight direction through the line of sight correction model, wherein the current line of sight direction points from the visual positioning point of the target user to the actual line of sight landing point, and the line of sight correction model is trained based on the aforementioned training method.

[0128] In some embodiments, considering the configuration requirements of the aforementioned sight line correction model, the aforementioned sight line correction system 500 may further include a model training module 540. Among them, the model training module 540 can be used to determine the user image of the target user when performing the interactive operation in response to the target user's interactive operation on the interactive control, wherein the user image includes the eye area of ​​the target user. Determine the actual optical axis direction of the target user when performing the interactive operation based on the user image, wherein the actual optical axis direction passes through the visual positioning point of the eye area in the user image and the eye center point of the target user. Determine the actual sight line landing point based on the interactive operation, wherein the actual sight line landing point includes the center position of the interactive control and / or the contact point of the target user on the interactive control. Determine the actual sight line direction of the target user when performing the interactive operation based on the actual sight line landing point and the visual positioning point, wherein the actual sight line direction points from the visual positioning point of the target user to the actual sight line landing point. Train the sight line correction model based on the actual sight line direction and the actual optical axis direction of the target user in the user image.

[0129] The present application also provides a vehicle, which includes the sight line correction system provided by the present application, and can execute the training model or sight line correction method provided by the present application. The vehicle provided by the present application can be a fully automatic driving vehicle or a semi-automatic driving vehicle. The specific model of the vehicle provided by the present application can be a car, a commercial vehicle, a truck, a bus, a tram, etc.

[0130] The present application also provides an electronic device, such as Figure 6 The electronic device 600 provided in the present application includes a memory 610, a processor 620, and an input / output interface 630. The memory 610, the processor 620, and the input / output interface 630 are connected through an internal connection path, the memory 610 is used to store instructions, and the processor 620 is used to execute the instructions stored in the memory 610 to control the input / output interface 630 to receive input data and information, and output data such as operation results.

[0131] It should be understood that in the embodiment of the present application, the processor 620 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the technical solution provided in the embodiment of the present application.

[0132] The memory 610 may include a read-only memory and a random access memory, and provides instructions and data to the processor 620. A portion of the processor 620 may also include a nonvolatile random access memory. For example, the processor 620 may also store information on the device type.

[0133] During the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 620 or an instruction in the form of software. The sight line correction method of the vehicle disclosed in the embodiment of the present application and the training method of its model can be directly embodied as a hardware processor execution, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 610, and the processor 620 reads the information in the memory 610 and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here. The present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction processor in the computer program product provided by the present application is executed, the sight line correction method of the vehicle provided by the present application and the training method of its model can be implemented.

[0134] Based on the same technical concept, the present application also provides a vehicle, comprising at least one processor, an in-vehicle sensor connected to the at least one processor in communication, and a memory connected to the at least one processor in communication. The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor implements the sight line correction method and the model training method thereof as described in any of the above embodiments based on the in-vehicle user image collected by the in-vehicle sensor.

[0135] Based on the same technical concept, the present application also provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction processor is executed, it implements the sight correction method and model training method thereof as described in any of the above embodiments.

[0136] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0137] The method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0138] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0139] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0140] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method for a sight line correction model, characterized in that: The sight line correction model is used to determine the sight line direction of the target user based on the optical axis direction of the target user in the image, and the training method includes: In response to the target user's interactive operation on the interactive control, determining a user image of the target user when performing the interactive operation, wherein the user image includes an eye area of ​​the target user; Determining, based on the user image, an actual optical axis direction of the target user when performing the interactive operation, wherein the actual optical axis direction passes through a visual positioning point of an eye region in the user image and an eye center point of the target user; Determine an actual sight point based on the interactive operation, wherein the actual sight point includes a center position of the interactive control and / or a contact point of the target user on the interactive control; Determine, based on the actual sight line landing point and the visual positioning point, the actual sight line direction of the target user when performing the interactive operation, wherein the actual sight line direction points from the visual positioning point of the target user to the actual sight line landing point; The sight line correction model is trained based on the actual sight line direction and the actual optical axis direction of the target user in the user image.

2. The training method according to claim 1, characterized in that: The training method further comprises: An optical axis determination model is trained based on the user image of the target user and the actual optical axis direction of the target user in the user image, wherein the optical axis determination model determines the optical axis direction of the user in the user image.

3. The training method according to claim 2, characterized in that: When the optical axis determination model has not completed training, determining the actual optical axis direction of the target user when performing the interactive operation based on the user image includes: Identifying a visual location point of the target user from the user image; Determine the eye space model of the target user based on the visual positioning point; Determining the eye center point of the target user from the eye space model; Determine the actual optical axis direction based on the eye center point and the visual positioning point; Training the optical axis determination model based on the user image of the target user and the actual optical axis direction of the target user in the user image until the optical axis determination model completes training; When the optical axis determination model has been trained, determining the actual optical axis direction of the target user when performing the interactive operation based on the user image includes: The visual positioning point is input into the optical axis determination model, and the actual optical axis direction of the target user is determined by the optical axis determination model.

4. The training method according to claim 1, characterized in that: The training of the sight line correction model based on the actual sight line direction and the actual optical axis direction of the target user in the user image includes: Training a plurality of candidate correction models based on the actual sight line direction and the actual optical axis direction of the target user in the user image; In response to completion of training of any candidate correction model, determining the accuracy of the candidate correction model; The sight line correction model is determined from the multiple candidate correction models based on the accuracy of each candidate correction model.

5. The training method according to claim 1, characterized in that: The training of the sight line correction model based on the actual sight line direction and the actual optical axis direction of the target user in the user image includes: Determining an estimated sight line direction based on the actual optical axis direction; Determining a confidence level of the interactive operation based on a difference between the estimated sight line direction and the actual sight line direction; In response to the confidence of the interactive operation being greater than a confidence threshold, the sight line correction model is trained based on the actual sight line direction of the target user in the user image and the actual optical axis direction.

6. A sight line correction method based on a sight line correction model, characterized in that: The method comprises: Collecting a user image of the target user, wherein the user image includes an eye area of ​​the target user; Determine the current optical axis direction of the target user based on the visual positioning point in the user image, wherein the current optical axis direction passes through the visual positioning point and the eye center point of the target user; The current optical axis direction is input into the line of sight correction model, and the current line of sight direction is determined by the line of sight correction model, wherein the current line of sight direction points from the visual positioning point of the target user to the actual line of sight landing point, and the line of sight correction model is trained based on the training method described in any one of claims 1 to 5.

7. The sight line correction method according to claim 6, characterized in that: The step of inputting the current optical axis direction into the sight line correction model and determining the current sight line direction by using the sight line correction model comprises: Calling the sight line correction model based on the identity information of the target user, wherein the sight line correction model is trained based on the data of the target user; Inputting the current optical axis direction into the sight line correction model, and determining the Kappa angle of the target user through the sight line correction model; The current optical axis direction is corrected based on the Kappa angle to determine the current line of sight direction.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the sight correction model described in any one of claims 1-5 and / or the sight correction method based on the sight correction model described in any one of claims 6 and 7.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction processor is executed, it implements the training method of the sight line correction model as described in any one of claims 1-5 and / or the sight line correction method based on the sight line correction model as described in any one of claims 6 and 7.

10. A vehicle, characterized in that: The vehicle comprises: at least one processor; an in-vehicle sensor communicatively coupled to the at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor implements the training method of the sight line correction model described in any one of claims 1 to 5 and / or the sight line correction method based on the sight line correction model described in any one of claims 6 and 7 based on the in-vehicle user image collected by the in-vehicle sensor.

Images