Automatic calibration method, device, vehicle, medium, equipment and program product

Abstract

The application discloses an automatic calibration method and device, a vehicle, a medium, equipment and a program product, and belongs to the field of data processing. The method comprises the following steps: converting the position coordinates of the nose tip point and the pupil center point of a face in a camera coordinate system into the coordinates of the nose tip point and the pupil center point of the face in an image coordinate, establishing a corresponding face orientation coordinate system and a line of sight orientation coordinate system, and obtaining a second face orientation and a second line of sight orientation in the corresponding coordinate system; and calibrating at least one of a face reference orientation and a line of sight reference orientation in the camera coordinate system by using at least one of a first face orientation and a first line of sight orientation of a predetermined number of frames of images and at least one of a second face orientation and a second line of sight orientation corresponding to the first face orientation and the first line of sight orientation. The application can achieve the purpose of automatically calibrating a driver at the initial stage of starting the vehicle, improve calibration efficiency, and calibrate a suitable coordinate system for different drivers.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to an automatic calibration method, apparatus, vehicle, medium, equipment, and program product. Background Technology

[0002] The Driver Monitor System (DMS) is implemented using a non-wide-angle camera on the driver's side and a wide-angle camera inside the vehicle for identification. After identification, the data is transmitted to the main control unit for fatigue detection. The fatigue detection main unit performs certain algorithm calculations based on the driver's actions, and the driver monitoring system transmits the driver's information to the human-machine interface for alarm reminders.

[0003] Applying driver monitoring systems to drivers and establishing a coordinate system suitable for each driver's individual circumstances can lay a solid foundation for future monitoring of driver behavior. Summary of the Invention

[0004] To achieve the above objectives, this application mainly provides an automatic calibration method, apparatus, vehicle, medium, equipment, and program product.

[0005] In a first aspect, embodiments of this application provide an automatic calibration method, which includes:

[0006] The camera parameters and images are input into the perception model. The perception model performs perception detection on the driver in the image under the camera coordinate system calibrated by the camera parameters, and obtains the driver's first face orientation based on the position coordinates of the nose tip and the first gaze orientation based on the position coordinates of the pupil center. The camera parameters include at least one of the camera extrinsic parameters and camera intrinsic parameters, and the pupil center is the center point of the line connecting the two pupil points.

[0007] The position coordinates of the nose tip and pupil center points in the camera coordinate system are transformed to the nose tip and pupil center point coordinates in the image coordinate system, respectively. At least one of the face orientation coordinate system and the gaze orientation coordinate system is established with at least one of the nose tip and pupil center point coordinates as its corresponding origin. At least one of the second face orientation in the face orientation coordinate system and the second gaze orientation in the gaze orientation coordinate system is obtained.

[0008] At least one of a first face orientation and a first gaze orientation is calibrated in the camera coordinate system using at least one of a first face orientation and a first gaze orientation, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation.

[0009] Optionally, by using at least one of a first face orientation and a first gaze orientation from a predetermined number of frame images, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation, at least one of the following is calibrated in the camera coordinate system:

[0010] Based on the radian values ​​of the first face orientation relative to its various three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​of the second face orientation relative to its various coordinate axes in the face orientation coordinate system, the face orientation offset between the first and second face orientations is determined; the face orientation offsets of a predetermined number of frames are weighted and calculated to obtain the average face orientation, and the average face orientation is used to calibrate and obtain the reference face orientation in the camera coordinate system; or

[0011] Based on the radian values ​​between the first gaze orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second gaze orientation and its respective coordinate axes in the gaze orientation coordinate system, the gaze orientation offset between the first gaze orientation and the second gaze orientation is determined; the face orientation offset of a predetermined number of frames is weighted and calculated to obtain the average gaze orientation, and the gaze reference orientation in the camera coordinate system is obtained by calibration using the average gaze orientation.

[0012] Optionally, the radian values ​​in the face orientation coordinate system and the gaze orientation coordinate system include at least one of the X-axis radian value, the Y-axis radian value, and the Z-axis radian value, respectively; the face orientation offset includes at least one of the X-axis face orientation offset, the Y-axis face orientation offset, and the Z-axis face orientation offset; and the gaze orientation offset includes at least one of the X-axis gaze orientation offset, the Y-axis gaze orientation offset, and the Z-axis gaze orientation offset.

[0013] Optionally, based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system, the face orientation offset between the first face orientation and its respective face orientation is determined, including: obtaining at least one of the corresponding X-axis radian values, Y-axis radian values, and Z-axis radian values ​​representing the distance between the first face orientation and its corresponding X-axis, Y-axis, and Z-axis in the camera coordinate system; obtaining the distance between the second face orientation and its respective face orientation in the face orientation coordinate system. At least one of the X-axis radian values, Y-axis radian values, and Z-axis radian values ​​respectively, which are distanced from its X-axis, Y-axis, and Z-axis; by calculating the radian offset of the corresponding X-axis radian value, Y-axis radian value, and Z-axis radian value in the camera coordinate system with the corresponding X-axis radian value, Y-axis radian value, and Z-axis radian value in the face orientation coordinate system, at least one of the X-axis face orientation offset, Y-axis face orientation offset, and Z-axis face orientation offset in the face orientation offset is obtained.

[0014] Optionally, at least one of the following coordinate systems can be established, with the nose tip coordinates and pupil center coordinates as the origin: using at least one of the nose tip coordinates and pupil center coordinates as the origin, respectively, to establish at least one of the following face orientation coordinate systems and gaze orientation coordinate systems based on the two-dimensional vertical plane of the image.

[0015] Secondly, embodiments of this application provide an automatic calibration device, which includes:

[0016] The orientation acquisition module is used to input camera parameters and images into the perception model. The perception model performs perception detection on the driver in the image under the camera coordinate system calibrated by the camera parameters, and obtains the driver's first face orientation based on the position coordinates of the nose tip and the first gaze orientation based on the position coordinates of the pupil center. The camera parameters include at least one of camera extrinsic parameters and camera intrinsic parameters, and the pupil center is the center point of the line connecting the two pupil points.

[0017] The orientation acquisition module is used to convert the position coordinates of the nose tip point and the pupil center point of the face in the camera coordinate system to the position coordinates of the nose tip point and the pupil center point of the face in the image coordinate system, respectively. It establishes at least one of the corresponding face orientation coordinate system and gaze orientation coordinate system with at least one of the coordinates of the nose tip point and the pupil center point as its corresponding origin, and obtains at least one of the second face orientation in the face orientation coordinate system and the first gaze orientation in the gaze orientation coordinate system.

[0018] An orientation calibration module is used to calibrate at least one of a face reference orientation and a gaze reference orientation in a camera coordinate system using at least one of a first face orientation and a first gaze orientation, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation.

[0019] Optionally, by using at least one of a first face orientation and a first gaze orientation from a predetermined number of frame images, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation, at least one of the following is calibrated in the camera coordinate system:

[0020] Based on the radian values ​​of the first face orientation relative to its various three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​of the second face orientation relative to its various coordinate axes in the face orientation coordinate system, the face orientation offset between the first and second face orientations is determined; the face orientation offsets of a predetermined number of frames are weighted and calculated to obtain the average face orientation, and the average face orientation is used to calibrate and obtain the reference face orientation in the camera coordinate system; or

[0021] Based on the radian values ​​between the first gaze orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second gaze orientation and its respective coordinate axes in the gaze orientation coordinate system, the gaze orientation offset between the first gaze orientation and the second gaze orientation is determined; the face orientation offset of a predetermined number of frames is weighted and calculated to obtain the average gaze orientation, and the gaze reference orientation in the camera coordinate system is obtained by calibration using the average gaze orientation.

[0022] Optionally, the radian values ​​in the face orientation coordinate system and the gaze orientation coordinate system include at least one of the X-axis radian value, the Y-axis radian value, and the Z-axis radian value, respectively; the face orientation offset includes at least one of the X-axis face orientation offset, the Y-axis face orientation offset, and the Z-axis face orientation offset; and the gaze orientation offset includes at least one of the X-axis gaze orientation offset, the Y-axis gaze orientation offset, and the Z-axis gaze orientation offset.

[0023] Optionally, based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system, the face orientation offset between the first face orientation and its respective face orientation is determined, including: obtaining at least one of the corresponding X-axis radian values, Y-axis radian values, and Z-axis radian values ​​representing the distance between the first face orientation and its corresponding X-axis, Y-axis, and Z-axis in the camera coordinate system; obtaining the distance between the second face orientation and its respective face orientation in the face orientation coordinate system. At least one of the X-axis radian values, Y-axis radian values, and Z-axis radian values ​​respectively, which are distanced from its X-axis, Y-axis, and Z-axis; by calculating the radian offset of the corresponding X-axis radian value, Y-axis radian value, and Z-axis radian value in the camera coordinate system with the corresponding X-axis radian value, Y-axis radian value, and Z-axis radian value in the face orientation coordinate system, at least one of the X-axis face orientation offset, Y-axis face orientation offset, and Z-axis face orientation offset in the face orientation offset is obtained.

[0024] Optionally, at least one of the following coordinate systems can be established, with the nose tip coordinates and pupil center coordinates as the origin: using at least one of the nose tip coordinates and pupil center coordinates as the origin, respectively, to establish at least one of the following face orientation coordinate systems and gaze orientation coordinate systems based on the two-dimensional vertical plane of the image.

[0025] Thirdly, embodiments of this application provide a vehicle, wherein the vehicle includes the automatic calibration device of embodiment two.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed, cause the computer to perform the automatic calibration method in Scheme 1.

[0027] Fifthly, embodiments of this application provide a computer device including at least one processor, the processor and a memory coupled together, the memory storing computer instructions, and the computer instructions, when executed by the processor, implementing the automatic calibration method in Scheme 1.

[0028] Sixthly, embodiments of this application provide a computer program product comprising computer instructions, wherein the computer instructions, when executed, implement the automatic calibration method as described in Scheme 1.

[0029] The technical solution of this application transforms the position coordinates of the nose tip and pupil center points of the face detected by the perception model in the camera coordinate system to the image coordinate system, and establishes a face orientation coordinate system and a gaze orientation coordinate system respectively. In this way, the face orientation and gaze orientation corresponding to the face orientation coordinate system and the gaze orientation coordinate system are obtained, and the face orientation and gaze orientation in the camera coordinate system are calibrated respectively to obtain the face reference orientation or gaze reference orientation. This achieves the purpose of automatically calibrating the driver's head at the beginning of vehicle startup, improving calibration efficiency, and tailoring a suitable coordinate system for different drivers. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description exemplarily illustrate some embodiments of this application.

[0031] Figure 1 This is a schematic diagram illustrating a specific implementation of an automatic calibration method according to this application;

[0032] Figure 2 This is a schematic diagram illustrating a specific embodiment of an automatic calibration device according to this application;

[0033] Figure 3 This is a schematic diagram illustrating a specific embodiment of a computer device according to this application.

[0034] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0035] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings, so that the advantages and features of this application can be more easily understood by those skilled in the art, thereby providing a clearer and more definite definition of the scope of protection of this application.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0037] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. The specific embodiments described below can be combined with each other to form new embodiments. The same or similar ideas or processes described in one embodiment may not be repeated in other embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0038] Figure 1 This paper illustrates a specific implementation of an automatic calibration method according to the present application.

[0039] exist Figure 1 In the specific implementation shown, an automatic calibration method mainly includes:

[0040] Step S101: Input the camera parameters and the image into the perception model. The perception model performs perception detection on the driver in the image under the camera coordinate system calibrated by the camera parameters, and obtains the driver's first face orientation based on the position coordinates of the nose tip and the first gaze orientation based on the position coordinates of the pupil center. The camera parameters include at least one of the camera extrinsic parameters and camera intrinsic parameters, and the pupil center is the center point of the line connecting the two pupil points.

[0041] In this embodiment, the perception model can directly sense and detect the face orientation and gaze orientation in the camera coordinate system. Direct output from the perception model makes this calibration scheme more efficient and lays the foundation for subsequent calculations.

[0042] In a specific example of this application, camera parameters are divided into extrinsic and intrinsic parameters. Extrinsic parameters are parameters in the world coordinate system, such as the camera's position and rotation direction; intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length and pixel size. By calibrating the camera coordinate system using both extrinsic and intrinsic parameters, it becomes easier to obtain the positions of the tip of the nose and the center of the pupil on a person's face.

[0043] exist Figure 1 In the specific embodiments shown, an automatic calibration method further includes:

[0044] Step S102: Convert the position coordinates of the nose tip and pupil center points of the face in the camera coordinate system to the position coordinates of the nose tip and pupil center points of the face in the image coordinate system, respectively. Establish at least one of the face orientation coordinate system and the gaze orientation coordinate system with at least one of the coordinates of the nose tip and pupil center points of the face as the origin, and obtain at least one of the second face orientation in the face orientation coordinate system and the second gaze orientation in the gaze orientation coordinate system.

[0045] In this embodiment, the coordinates of the nose tip of the face in the camera coordinate system are transformed to the coordinates of the nose tip of the face in the image coordinate system, and a face orientation coordinate system is established with the nose tip of the face as the origin to obtain the second face orientation in the face orientation coordinate system, which lays the data foundation for calibrating the first face orientation in the camera coordinate system; the coordinates of the pupil center point in the camera coordinate system are transformed to the coordinates of the pupil center point in the image coordinate system, and a gaze orientation coordinate system is established with the pupil center point as the origin to obtain the second gaze orientation in the gaze orientation coordinate system, which lays the data foundation for calibrating the first gaze orientation in the camera coordinate system.

[0046] Specifically, during the initial period after vehicle startup, there may be situations where drivers change shifts or return after rest. In these cases, it's necessary to calibrate the three-dimensional coordinate system of the driver's head. Under the camera coordinate system calibrated with camera parameters, the perception model detects the position coordinates of various facial key points in the image, obtaining the position coordinates of the tip of the nose and the two pupils. Through complex calculations, it also obtains the position coordinates of the pupil center point, located at the center of the line connecting the two pupils. Furthermore, based on the position coordinates of the tip of the nose and the orientation of the nostrils, the perception model derives the facial orientation based on the tip of the nose; and based on the position coordinates of the two pupils and the direction of the gaze, it derives the gaze direction based on the pupil center point.

[0047] In one specific embodiment of this application, establishing at least one of the corresponding face orientation coordinate system and gaze orientation coordinate system with at least one of the coordinates of the nose tip point and the pupil center point as the origin includes: establishing at least one of the face orientation coordinate system and gaze orientation coordinate system based on the two-dimensional vertical plane of the image with at least one of the coordinates of the nose tip point and the pupil center point as the origin.

[0048] In this embodiment, a face orientation coordinate system is established on the vertical plane of the image, with the coordinates of the tip of the face as the origin. This is a three-dimensional coordinate system used to calibrate the face orientation detected by the perception model in the camera coordinate system. Similarly, a gaze orientation coordinate system is established on the vertical plane of the image, with the coordinates of the pupil center as the origin. This is also a three-dimensional coordinate system used to calibrate the gaze orientation detected by the perception model in the camera coordinate system. This not only provides a suitable three-dimensional coordinate system tailored to the driver's head but also lays the foundation for calibrating the driver's face orientation and gaze orientation.

[0049] It should be noted that the orientation of the face and the direction of the gaze can be either an angle or an arc, depending on the nature of the three-dimensional coordinate system established. This application does not impose any specific restrictions.

[0050] exist Figure 1 In the specific embodiments shown, an automatic calibration method further includes:

[0051] Step S103: Using at least one of the first face orientation and the first gaze orientation of a predetermined number of frame images, and at least one of the second face orientation and the second gaze orientation corresponding to the first face orientation and the first gaze orientation, at least one of the face reference orientation and the gaze reference orientation is calibrated in the camera coordinate system.

[0052] In this embodiment, the facial reference orientation is calibrated by the first facial orientation and the second facial orientation, and the visual reference orientation is calibrated by the first visual orientation and the second visual orientation; the facial orientation and visual orientation are calibrated according to the different body shapes of the drivers, thereby improving calibration efficiency and making the calibration process simple and efficient.

[0053] It should be noted that to ensure the accuracy of facial and gaze reference orientations, a predetermined number of frames must generally be acquired. The value of this predetermined number is set according to the specific scenario. For example, in scenarios involving long-distance driving, based on practical experience, it is necessary to acquire a sufficient number of images within 200 seconds to calibrate the driver's face and gaze. At an acquisition rate of 10 frames per second, the predetermined number within 200 seconds would be 2000 frames.

[0054] In one specific embodiment of this application, at least one of a first face orientation and a first gaze orientation in a camera coordinate system is calibrated using at least one of a first face orientation and a first gaze orientation in a predetermined number of frame images, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation. This includes at least one of the following: determining the face orientation offset between the first face orientation and the second face orientation based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system; and for a predetermined number of frames... The face orientation offset of a predetermined number of frames of images is weighted and calculated to obtain the average face orientation. The average face orientation is then used to calibrate and obtain the reference face orientation in the camera coordinate system. Alternatively, the gaze orientation offset between the first gaze orientation and the second gaze orientation is determined based on the radian values ​​between the first gaze orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second gaze orientation and its respective coordinate axes in the gaze orientation coordinate system. The face orientation offset of a predetermined number of frames of images is weighted and calculated to obtain the average gaze orientation. The average gaze orientation is then used to calibrate and obtain the reference gaze orientation in the camera coordinate system.

[0055] In this embodiment, the first step is to determine the radian values ​​of the first face orientation and the first gaze orientation in the three dimensions of the camera coordinate system. These values ​​are then compared with the radian values ​​of the second face orientation and the second gaze orientation in the three dimensions of their corresponding coordinate systems to obtain the offset of the face orientation and the offset of the gaze orientation. By calculating the average of the offsets, the calibrated reference orientation is obtained. This method is simple and efficient, making the calibration process of the driver's face and gaze more convenient and faster.

[0056] In a specific example of this application, the face orientation offset is weighted and calculated using the covariance formula, resulting in an average face orientation value that includes the average across three coordinate axes. Similarly, the gaze orientation offset is weighted and calculated using the covariance formula, resulting in an average gaze orientation value that also includes the average across three coordinate axes. By correcting the corresponding face orientation and gaze orientation in the camera coordinate system using the average face orientation and average gaze orientation values ​​respectively, the calibration efficiency is improved and the process is simplified.

[0057] In one specific embodiment of this application, the radian values ​​in the face orientation coordinate system and the gaze orientation coordinate system respectively include at least one of the X-axis radian value, the Y-axis radian value, and the Z-axis radian value; the face orientation offset includes at least one of the X-axis face orientation offset, the Y-axis face orientation offset, and the Z-axis face orientation offset; and the gaze orientation offset includes at least one of the X-axis gaze orientation offset, the Y-axis gaze orientation offset, and the Z-axis gaze orientation offset.

[0058] In this embodiment, the face orientation coordinate system and the gaze orientation coordinate system each have three coordinate axes: the X-axis, the Y-axis, and the Z-axis. The X-axis radian value includes the pitch angle (rotation around the X-axis), the Y-axis radian value includes the yaw angle (rotation around the Y-axis), and the Z-axis radian value includes the roll angle (rotation around the Z-axis). This provides the data basis for calculating the offset. The face orientation offset and the gaze orientation offset each contain offsets in three coordinate dimensions, making subsequent orientation calibration more accurate.

[0059] In one specific embodiment of this application, determining the face orientation offset between the first face orientation and the second face orientation based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system, includes: obtaining at least one of the corresponding X-axis radian values, Y-axis radian values, and Z-axis radian values ​​representing the distance between the first face orientation and its corresponding X-axis, Y-axis, and Z-axis in the camera coordinate system; obtaining the radian values ​​between the first face orientation and its respective X-axis, Y-axis, and Z-axis in the face orientation coordinate system; and obtaining the radian values ​​between the first face orientation and its respective X-axis, Y-axis, and Z-axis radian values ​​in the face orientation coordinate system. The distances of the two faces from their respective X-axis, Y-axis, and Z-axis radian values ​​are calculated using at least one of the following coordinate axis radian values: X-axis face orientation offset, Y-axis face orientation offset, and Z-axis face orientation offset.

[0060] In this embodiment, the distance between the radian values ​​on the coordinate axes of the same dimension under different coordinate systems is calculated to obtain the offset under the corresponding coordinate axis. The orientation coordinate under the corresponding dimension is corrected by the offset under each coordinate axis, which helps to improve the accuracy of face orientation and gaze orientation.

[0061] In a specific example of this application, the three-dimensional coordinate system of the driver's head is automatically calibrated based on possible driver shift changes or driver return after rest during the initial period of vehicle startup. If there is a driver shift change, each driver has different height and build, so the position and direction of the driver's face and gaze may not be exactly the same, so it is necessary to recalibrate the driver's head.

[0062] In a specific example of this application, using a camera coordinate system calibrated with camera parameters, the perception model detects the position coordinates of key facial points of the driver in the image within the camera coordinate system. This allows it to obtain the position coordinates of the tip of the nose and the center point of the pupil (the point between the two pupils). The perception model also detects the facial orientation based on the nose tip and nostril orientation; and the gaze orientation based on the pupil center point. The direct output of the perception model makes this calibration scheme more efficient and lays the foundation for subsequent calculations. The position coordinates of the nose tip and pupil center point in the camera coordinate system are then transformed... The coordinates in the image coordinate system are used to lay the foundation for the calibration of the driver's head coordinate system. The face orientation coordinate system and the gaze orientation coordinate system are established with the coordinates in the image coordinate system as the origin. The offset of face orientation and gaze orientation in the three coordinate dimensions of the camera coordinate system is calculated by comparing them with the offsets in the three coordinate dimensions of the face orientation coordinate system and the gaze orientation in the three coordinate dimensions of the gaze orientation coordinate system. The offsets are then weighted using the covariance formula, and the average value obtained by the weighted calculation is used to calibrate the face orientation and gaze orientation in the camera coordinate system. The whole process is simple and efficient, and can efficiently calibrate for drivers of different body shapes according to actual conditions.

[0063] Figure 2 This paper illustrates a specific embodiment of an automatic calibration device according to the present application.

[0064] exist Figure 2 In the specific embodiment shown, an automatic calibration device mainly includes:

[0065] The orientation acquisition module 201 is used to input camera parameters and images into the perception model. The perception model performs perception detection on the driver in the image under the camera coordinate system calibrated by the camera parameters, and obtains the driver's first face orientation based on the position coordinates of the nose tip and the first gaze orientation based on the position coordinates of the pupil center. The camera parameters include at least one of camera extrinsic parameters and camera intrinsic parameters, and the pupil center is the center point of the line connecting the two pupil points.

[0066] The orientation acquisition module 202 is used to convert the position coordinates of the nose tip point and the position coordinates of the pupil center point in the camera coordinate system to the nose tip point coordinates and the pupil center point coordinates in the image coordinate system, respectively, and establish at least one of the corresponding face orientation coordinate system and gaze orientation coordinate system with at least one of the nose tip point coordinates and the pupil center point coordinates as the origin, and obtain at least one of the second face orientation in the face orientation coordinate system and the first gaze orientation in the gaze orientation coordinate system.

[0067] Orientation calibration module 203 is used to calibrate at least one of a first face orientation and a first gaze orientation in the camera coordinate system using at least one of a first face orientation and a first gaze orientation, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation.

[0068] In this embodiment, the perception model of the orientation acquisition module 201 can directly perceive and detect the face orientation and gaze orientation in the camera coordinate system. The direct output of the perception model makes this calibration scheme more efficient and lays the foundation for subsequent calculations. The orientation acquisition module 202 transforms the coordinates of the nose tip of the face in the camera coordinate system to the coordinates of the nose tip of the face in the image coordinate system, and establishes a face orientation coordinate system with it as the origin to obtain the second face orientation in the face orientation coordinate system, laying the data foundation for calibrating the first face orientation in the camera coordinate system. The module also transforms the coordinates of the pupil center point in the camera coordinate system to the coordinates of the pupil center point in the image coordinate system, and establishes a gaze orientation coordinate system with it as the origin to obtain the second gaze orientation in the gaze orientation coordinate system, laying the data foundation for calibrating the first gaze orientation in the camera coordinate system. The orientation calibration module 203 calibrates the face reference orientation using the first face orientation and the second face orientation, and calibrates the gaze reference orientation using the first gaze orientation and the second gaze orientation. The module can tailor the face orientation and gaze orientation for drivers of different body types, improving calibration efficiency. The calibration process is simple and efficient.

[0069] In one specific embodiment of this application, establishing at least one of the corresponding face orientation coordinate system and gaze orientation coordinate system with at least one of the coordinates of the nose tip point and the pupil center point as the origin includes: establishing at least one of the face orientation coordinate system and gaze orientation coordinate system based on the two-dimensional vertical plane of the image with at least one of the coordinates of the nose tip point and the pupil center point as the origin.

[0070] In this embodiment, a face orientation coordinate system is established on the vertical plane of the image, with the coordinates of the tip of the face as the origin. This is a three-dimensional coordinate system used to calibrate the face orientation detected by the perception model in the camera coordinate system. Similarly, a gaze orientation coordinate system is established on the vertical plane of the image, with the coordinates of the pupil center as the origin. This is also a three-dimensional coordinate system used to calibrate the gaze orientation detected by the perception model in the camera coordinate system. This not only provides a suitable three-dimensional coordinate system tailored to the driver's head but also lays the foundation for calibrating the driver's face orientation and gaze orientation.

[0071] In one specific embodiment of this application, at least one of a first face orientation and a first gaze orientation in a camera coordinate system is calibrated using at least one of a first face orientation and a first gaze orientation in a predetermined number of frame images, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation. This includes at least one of the following: determining the face orientation offset between the first face orientation and the second face orientation based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system; and for a predetermined number of frames... The face orientation offset of a predetermined number of frames of images is weighted and calculated to obtain the average face orientation. The average face orientation is then used to calibrate and obtain the reference face orientation in the camera coordinate system. Alternatively, the gaze orientation offset between the first gaze orientation and the second gaze orientation is determined based on the radian values ​​between the first gaze orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second gaze orientation and its respective coordinate axes in the gaze orientation coordinate system. The face orientation offset of a predetermined number of frames of images is weighted and calculated to obtain the average gaze orientation. The average gaze orientation is then used to calibrate and obtain the reference gaze orientation in the camera coordinate system.

[0072] In this embodiment, the first step is to determine the radian values ​​of the first face orientation and the first gaze orientation in the three dimensions of the camera coordinate system. These values ​​are then compared with the radian values ​​of the second face orientation and the second gaze orientation in the three dimensions of their corresponding coordinate systems to obtain the offset of the face orientation and the offset of the gaze orientation. By calculating the average of the offsets, the calibrated reference orientation is obtained. This method is simple and efficient, making the calibration process of the driver's face and gaze more convenient and faster.

[0073] In one specific embodiment of this application, the radian values ​​in the face orientation coordinate system and the gaze orientation coordinate system respectively include at least one of the X-axis radian value, the Y-axis radian value, and the Z-axis radian value; the face orientation offset includes at least one of the X-axis face orientation offset, the Y-axis face orientation offset, and the Z-axis face orientation offset; and the gaze orientation offset includes at least one of the X-axis gaze orientation offset, the Y-axis gaze orientation offset, and the Z-axis gaze orientation offset.

[0074] In this embodiment, the face orientation coordinate system and the gaze orientation coordinate system each have three coordinate axes: the X-axis, the Y-axis, and the Z-axis. The X-axis radian value includes the pitch angle (rotation around the X-axis), the Y-axis radian value includes the yaw angle (rotation around the Y-axis), and the Z-axis radian value includes the roll angle (rotation around the Z-axis). This provides the data basis for calculating the offset. The face orientation offset and the gaze orientation offset each contain offsets in three coordinate dimensions, making subsequent orientation calibration more accurate.

[0075] In one specific embodiment of this application, determining the face orientation offset between the first face orientation and the second face orientation based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system, includes: obtaining at least one of the corresponding X-axis radian values, Y-axis radian values, and Z-axis radian values ​​representing the distance between the first face orientation and its corresponding X-axis, Y-axis, and Z-axis in the camera coordinate system; obtaining the radian values ​​between the first face orientation and its respective X-axis, Y-axis, and Z-axis in the face orientation coordinate system; and obtaining the radian values ​​between the first face orientation and its respective X-axis, Y-axis, and Z-axis radian values ​​in the face orientation coordinate system. The distances of the two faces from their respective X-axis, Y-axis, and Z-axis radian values ​​are calculated using at least one of the following coordinate axis radian values: X-axis face orientation offset, Y-axis face orientation offset, and Z-axis face orientation offset.

[0076] In this embodiment, the distance between the radian values ​​on the coordinate axes of the same dimension under different coordinate systems is calculated to obtain the offset under the corresponding coordinate axis. The orientation coordinate under the corresponding dimension is corrected by the offset under each coordinate axis, which helps to improve the accuracy of face orientation and gaze orientation.

[0077] The automatic calibration device provided in this application can be used to perform the automatic calibration method described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0078] In one specific embodiment of this application, the functional modules in the automatic calibration device proposed in this application can be directly in hardware, in software modules executed by a processor, or in a combination of both.

[0079] Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in this art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium.

[0080] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, but alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium can reside as discrete components in the user terminal.

[0081] In another specific embodiment of this application, a vehicle is provided, wherein the vehicle includes the automatic calibration device of any embodiment. Optionally, the vehicle includes a processor and a memory, the processor and memory being coupled, the vehicle being used to implement the appendix to this application specification. Figure 1 The automatic calibration method in any of the embodiments shown.

[0082] In another specific embodiment of this application, a computer-readable storage medium is provided, which stores computer instructions that, when executed, cause a computer to perform the automatic calibration method in any embodiment.

[0083] Figure 3 A specific embodiment of a computer device according to this application is shown.

[0084] exist Figure 3 In the specific embodiments shown, a computer device includes at least one processor, the processor and a memory coupled together, the memory storing computer instructions, which, when executed by the processor, implement the automatic calibration method in any embodiment.

[0085] In another specific embodiment of this application, a computer program product includes computer instructions, wherein the computer instructions, when executed, implement the automatic calibration method as in any embodiment.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An automatic calibration method, characterized in that, The automatic calibration method is applied to an automatic calibration device, including: Camera parameters and images are input into a perception model. The perception model, calibrated using the camera parameters, performs perception detection on the driver in the image, obtaining the driver's first facial orientation based on the coordinates of the nose tip and the first gaze orientation based on the coordinates of the pupil center. The camera parameters include at least one of camera extrinsic and intrinsic parameters, and the pupil center is the center point of the line connecting two pupil points. The coordinates of the nose tip and the pupil center in the camera coordinate system are transformed to the nose tip and pupil center coordinates in the image coordinate system, respectively. At least one of the nose tip and pupil center coordinates is established as the origin, corresponding to at least one of the facial orientation coordinate system and the gaze orientation coordinate system. At least one of the second facial orientation in the facial orientation coordinate system and the second gaze orientation in the gaze orientation coordinate system is obtained. Specifically, the pupil center coordinates in the camera coordinate system are transformed to the pupil center coordinates in the image coordinate system, and this coordinate is used as the origin to establish the gaze orientation coordinate system, obtaining the second gaze orientation in the gaze orientation coordinate system. At least one of the first face orientation and the first gaze orientation in the camera coordinate system is calibrated using at least one of the first face orientation and the first gaze orientation in a predetermined number of frames of the image, and at least one of the second face orientation and the second gaze orientation corresponding to the first face orientation and the first gaze orientation.

2. The automatic calibration method as described in claim 1, characterized in that, The step of calibrating at least one of the first face orientation and the first gaze orientation in the camera coordinate system using at least one of the first face orientation and the first gaze orientation, and at least one of the second face orientation and the second gaze orientation corresponding to the first face orientation and the first gaze orientation, includes at least one of the following: Based on the radian values ​​of the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​of the second face orientation and its respective coordinate axes in the face orientation coordinate system, the face orientation offset between the first face orientation and the second face orientation is determined; The face orientation offset of the predetermined number of frames of the image is weighted and calculated to obtain the average face orientation, and the face reference orientation in the camera coordinate system is obtained by calibration using the average face orientation. or Based on the radian values ​​between the first line of sight and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second line of sight and its respective coordinate axes in the line of sight coordinate system, the line of sight offset between the first line of sight and the second line of sight is determined; The face orientation offset of the predetermined number of frames of the image is weighted and calculated to obtain the average gaze orientation, and the gaze reference orientation in the camera coordinate system is obtained by calibration using the average gaze orientation.

3. The automatic calibration method as described in claim 2, characterized in that, The radian values ​​in the face orientation coordinate system and the gaze orientation coordinate system respectively include at least one of the X-axis radian value, Y-axis radian value and Z-axis radian value; The face orientation offset includes at least one of the X-axis face orientation offset, Y-axis face orientation offset, and Z-axis face orientation offset; and The line-of-sight orientation offset includes at least one of the X-axis line-of-sight orientation offset, the Y-axis line-of-sight orientation offset, and the Z-axis line-of-sight orientation offset.

4. The automatic calibration method as described in claim 3, characterized in that, The step of determining the face orientation offset between the first face orientation and the second face orientation based on the radian values ​​between the first face orientation and its respective three-dimensional coordinate axes in the camera coordinate system, and the radian values ​​between the second face orientation and its respective coordinate axes in the face orientation coordinate system, includes: In the camera coordinate system, obtain at least one of the X-axis radian values, Y-axis radian values, and Z-axis radian values ​​corresponding to the distance between the first face orientation and its corresponding X-axis, Y-axis, and Z-axis. Obtain at least one of the following values ​​in radians (X-axis, Y-axis, and Z-axis) corresponding to the distances of the second face orientation from its X-axis, Y-axis, and Z-axis, respectively, in the face orientation coordinate system: By calculating the corresponding coordinate axis radian offset between at least one of the X-axis radian value, Y-axis radian value, and Z-axis radian value in the camera coordinate system and at least one of the X-axis radian value, Y-axis radian value, and Z-axis radian value in the face orientation coordinate system, at least one of the X-axis face orientation offset, Y-axis face orientation offset, and Z-axis face orientation offset in the face orientation offset is obtained.

5. The automatic calibration method as described in claim 1, characterized in that, The step of establishing at least one of the following coordinate systems—a face orientation coordinate system and a gaze orientation coordinate system—with at least one of the coordinates of the nose tip point and the pupil center point as its origin includes: Using at least one of the coordinates of the nose tip of the face and the center of the pupil as its origin, establish at least one of the face orientation coordinate system and the gaze orientation coordinate system on the two-dimensional vertical plane of the image.

6. An automatic calibration device, characterized in that, include: The orientation acquisition module is used to input camera parameters and images into the perception model. The perception model performs perception detection on the driver in the image under the camera coordinate system calibrated by the camera parameters to obtain the driver's first face orientation based on the position coordinates of the nose tip and the first gaze orientation based on the position coordinates of the pupil center. The camera parameters include at least one of camera extrinsic parameters and camera intrinsic parameters, and the pupil center is the center point of the line connecting the two pupil points. The orientation acquisition module is used to convert the position coordinates of the nose tip point and the pupil center point in the camera coordinate system to the nose tip point and pupil center point coordinates in the image coordinate system, respectively. It establishes at least one of the corresponding face orientation coordinate system and gaze orientation coordinate system with at least one of the nose tip point coordinates and pupil center point coordinates as the origin. It obtains at least one of the second face orientation in the face orientation coordinate system and the first gaze orientation in the gaze orientation coordinate system. Specifically, the pupil center point position coordinates in the camera coordinate system are converted to the pupil center point coordinates in the image coordinate system, and the gaze orientation coordinate system is established with the pupil center point as the origin to obtain the second gaze orientation in the gaze orientation coordinate system. An orientation calibration module is used to calibrate at least one of a face reference orientation and a gaze reference orientation in the camera coordinate system by using at least one of a first face orientation and a first gaze orientation of a predetermined number of frames of the image, and at least one of a second face orientation and a second gaze orientation corresponding to the first face orientation and the first gaze orientation.

7. A vehicle, characterized in that, The vehicle includes the automatic calibration device as described in claim 6.

8. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed, the computer performs the automatic calibration method as described in any one of claims 1-5.

9. A computer device comprising at least one processor, said processor coupled to a memory storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the automatic calibration method as described in any one of claims 1-5.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed, they implement the automatic calibration method as described in any one of claims 1-5.

Images