A calibration method and apparatus for a vehicle-mounted camera, a computer device, and a storage medium.

Abstract

This disclosure provides a calibration method and apparatus for an in-vehicle camera, a computer device, and a storage medium. The method includes: acquiring first position information of the current position of the in-vehicle camera relative to a reference position; wherein the in-vehicle camera is mounted on a steering column; determining a first transformation matrix of the current position relative to the reference position based on the first position information of the in-vehicle camera; acquiring third position information of the reference position in a vehicle cabin coordinate system and rotation angle information of the steering column in the reference position; and determining a second transformation matrix of the current position of the in-vehicle camera relative to the vehicle cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information. This scheme enables the calibration of in-vehicle cameras with variable positions, thereby improving the accuracy of driver status assessment in systems such as Driver Monitoring Systems (DMS).

Description

Technical Field

[0001] This disclosure relates to the field of camera technology, and in particular to a calibration method and apparatus for a vehicle-mounted camera, a computer device, and a storage medium. Background Technology

[0002] As different brands and models pursue aesthetic enhancements and personalization in their vehicle interiors, the installation locations of in-vehicle cameras are becoming increasingly flexible, shifting from fixed to variable positions within the cabin. However, how to apply these variable-position in-vehicle cameras to systems such as Driver Monitoring Systems (DMS) to determine the driver's condition remains to be further explored. Summary of the Invention

[0003] This disclosure aims to provide a calibration method and apparatus for vehicle-mounted cameras, a computer device, and a storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a calibration method for a vehicle-mounted camera, the method comprising:

[0005] Acquire first position information of the current position of the vehicle-mounted camera relative to a reference position; wherein, the vehicle-mounted camera is mounted on the steering column;

[0006] Based on the first position information of the vehicle-mounted camera, a first transformation matrix is ​​determined relative to the reference position of the current position;

[0007] Obtain the third position information of the reference position in the vehicle cabin coordinate system and the rotation angle information of the steering column in the reference position;

[0008] Based on the first transformation matrix, the third position information, and the rotation angle information, a second transformation matrix is ​​determined for the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system.

[0009] Secondly, embodiments of this disclosure provide a calibration device for a vehicle-mounted camera, the device comprising:

[0010] The first acquisition module is used to acquire first position information of the current position of the vehicle-mounted camera relative to a reference position; wherein, the vehicle-mounted camera is mounted on the steering column;

[0011] The first determining module is used to determine a first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera;

[0012] The second acquisition module is used to acquire the third position information of the reference position in the vehicle cabin coordinate system and the rotation angle information of the steering column in the reference position;

[0013] The second determining module is used to determine the second transformation matrix of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information.

[0014] Thirdly, embodiments of this disclosure provide a computer device, including: a processor; and a memory for storing processor-executable instructions;

[0015] The processor is configured to perform the method described in the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0017] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0018] This disclosure mounts an in-vehicle camera on the steering column and presets a reference position as an intermediate term. It uses a two-step approach: first, it maps the current position of the in-vehicle camera to the reference position; then, based on the reference position to the third position information in the vehicle cabin coordinate system and the rotation angle information of the steering column in the reference position, it completes the transformation of the current position of the in-vehicle camera relative to the vehicle cabin coordinate system. This enables the calibration of the in-vehicle camera with variable position according to this disclosure, thereby improving the accuracy of driver status assessment in systems such as DMS.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a calibration method for a vehicle-mounted camera according to an embodiment of this disclosure;

[0021] Figure 2 This is an example diagram illustrating the mechanical information of a steering column according to an embodiment of this disclosure;

[0022] Figure 3 and Figure 4 This is an example diagram of the three-dimensional data of the rotation center of the steering column in the vehicle cabin coordinate system in an embodiment of this disclosure.

[0023] Figure 5 and Figure 6 This is an example diagram of the three-dimensional data of the reference position in the vehicle cabin coordinate system in the embodiments of this disclosure;

[0024] Figure 7 This is a schematic diagram illustrating a calibration method for a vehicle-mounted camera according to an embodiment of this disclosure.

[0025] Figure 8An example diagram of a camera calibration device provided in an embodiment of this disclosure is shown;

[0026] Figure 9 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure. Detailed Implementation

[0027] The technical solution of this disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] This disclosure provides a camera calibration method, the execution subject of which can be a camera calibration device, which can be a computer device such as a server or laptop, or an in-vehicle device. The in-vehicle device can be a vehicle-mounted system within the vehicle cabin, or a host device within the vehicle that can perform image and other data processing operations; this disclosure does not limit this specific method.

[0029] Figure 1 This is a flowchart illustrating a calibration method for a vehicle-mounted camera according to an embodiment of this disclosure, as shown below. Figure 1 As shown, the calibration method for vehicle-mounted cameras includes the following steps:

[0030] S11. Obtain the first position information of the current position of the vehicle-mounted camera relative to the reference position; wherein, the vehicle-mounted camera is mounted on the steering column;

[0031] S12. Based on the first position information of the vehicle-mounted camera, determine the first transformation matrix of the current position relative to the reference position;

[0032] S13. Obtain the third position information of the reference position in the vehicle cabin coordinate system and the rotation angle information of the steering column in the reference position;

[0033] S14. Based on the first transformation matrix, the third position information, and the rotation angle information, determine the second transformation matrix of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system.

[0034] In this embodiment, the steering wheel in the vehicle can be manually operated to change its height and rotation angle. These changes are based on the variable length and rotation angle of the steering column. For example, the position of the steering wheel along the length of the column changes due to variations in its length, and the steering wheel can rotate relative to its rotation center point based on variations in the column's rotation angle. When the vehicle-mounted camera is located on the steering column, its position changes based on the position of the steering column.

[0035] In this embodiment of the disclosure, a reference position for the vehicle-mounted camera is preset. The reference position can refer to the position of the steering column at a preset length and a preset rotation angle. The preset length can be the length of the column when the vehicle leaves the factory, and the preset rotation angle can be the angle of the column when the steering wheel is in a specified position, such as an angle of 45 degrees with the bottom plane of the vehicle cabin.

[0036] In step S11, the camera calibration device acquires the first position information of the vehicle camera at the current position relative to the reference position. The current position can be determined based on the position of the steering column during vehicle operation.

[0037] In step S12, the camera calibration device determines the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera. Assuming the transformation matrix of the current position relative to the reference position is M, multiplying the current position by M yields the reference position; that is, the first transformation matrix is ​​a mapping matrix from the current position to the reference position.

[0038] In this embodiment, since the position of the vehicle-mounted camera is variable, its position in the vehicle cabin coordinate system needs to be calibrated after the camera's position changes in order to analyze the in-vehicle situation based on the in-vehicle images captured by the camera. The length of the steering column when the camera is in its current position may differ from the length when the camera is in its reference position, and the rotation angle of the steering column when the camera is in its current position may also differ from the rotation angle when the camera is in its reference position. Therefore, it can be understood that the first transformation matrix includes translation and rotation information of the current position relative to the reference position.

[0039] In step S13, the camera calibration device acquires the third position information of the reference position in the vehicle cabin coordinate system and the rotation angle information of the steering column in the reference position. The vehicle cabin coordinate system is a pre-established vehicle cabin coordinate system with a fixed position in the vehicle cabin as the origin. The origin of the vehicle cabin is, for example, the place where the glasses case is placed in the vehicle cabin, or the center position of the vehicle central control display screen, etc., and this embodiment of the disclosure is not limited.

[0040] In this disclosure, the third position of the reference position relative to the vehicle cabin coordinate system is mechanical design information. For different vehicle models, the position information of the reference position in the vehicle cabin coordinate system (i.e., the third position information) and the rotation angle information of the reference position relative to the steering column may differ. The camera calibration device of this disclosure can directly read the third position information and the rotation angle information of the steering column in the reference position for the corresponding vehicle model.

[0041] In step S14, the camera calibration device determines the second transformation matrix of the vehicle-mounted camera's current position relative to the vehicle cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information. In this embodiment, assuming the second matrix of the vehicle-mounted camera's current position relative to the vehicle cabin coordinate system is E, multiplying the vehicle-mounted camera's current position by E yields the position of the vehicle-mounted camera in the vehicle cabin coordinate system. That is, the second transformation matrix is ​​the mapping matrix from the vehicle-mounted camera's current position to the vehicle cabin coordinate system.

[0042] If the vehicle-mounted camera, based on an embodiment of this disclosure, detects the driver's state, for example, by detecting the driver's gaze area, the positions of the pupils in the driver's face image captured by the vehicle-mounted camera at the current position can be located first. Then, for example, based on the intrinsic parameters of the vehicle-mounted camera and the calibration matrix determined by this disclosure, the positions of the pupil centers of the left and right eyes in the vehicle's coordinate system can be determined, thereby detecting the driver's gaze direction, determining the landing point of the gaze direction within the vehicle, and obtaining the driver's gaze area. For another example, detecting actions such as nodding, gestures, and blinking requires analyzing the acquired video using a fixed coordinate system to detect the corresponding actions. Taking nodding as an example, after detecting the position of the head in the image frame of the video, multiplying by the camera's intrinsic parameter matrix yields the position of the head in the camera coordinate system of the vehicle-mounted camera at the current position. Then, multiplying by the second transformation matrix of this disclosure gives the position of the head within the vehicle. The position of the head within the vehicle determined based on multiple consecutive frames of images can then determine whether the driver is nodding. If the driver is nodding, it can be determined that the driver is fatigued, and prompts may need to be output to encourage safe driving.

[0043] In related technologies, the location of vehicle-mounted cameras is fixed, for example, mounted on the "A" pillar, which is the connecting pillar between the windshield and the left and right front doors. Cameras mounted on the "A" pillar mostly capture the driver's side profile. Therefore, when analyzing the driver's facial expressions or behavior based on the acquired side profile images in a DMS system, errors are prone to occur, resulting in poor accuracy in assessing the driver's condition. Furthermore, vehicle-mounted cameras located on the "A" pillar also affect the aesthetics of the vehicle interior.

[0044] In contrast, this disclosure mounts the vehicle-mounted camera on the steering column, minimizing interference with occupants. Furthermore, by using a preset reference position as an intermediary, a two-step process is employed: first, the current position of the vehicle-mounted camera is mapped to the reference position; then, based on the reference position's third position information in the vehicle cabin coordinate system and the steering column's rotation angle information at the reference position, the transformation of the vehicle-mounted camera's current position relative to the vehicle cabin coordinate system is completed. This allows for the calibration of the variable-position vehicle-mounted camera, thereby improving the accuracy of driver status assessment in systems such as DMS.

[0045] In some embodiments, obtaining the first position information of the vehicle-mounted camera at its current position relative to a reference position includes:

[0046] The mechanical signals of the steering column are read to obtain the extension length data and deflection angle data of the steering column, which are used as the first position information.

[0047] In this embodiment of the disclosure, the extension length data of the steering column is the length of the steering column corresponding to the current position of the vehicle camera, and the length change data relative to the preset length of the steering column corresponding to the reference position of the vehicle camera; the deflection angle data of the steering column is the angle of the steering column corresponding to the current position of the vehicle camera, and the angle change data relative to the preset rotation angle of the steering column corresponding to the reference position of the vehicle camera.

[0048] In some embodiments, when the camera calibration device of this disclosure acquires first position information based on mechanical signals from the steering column, the mechanical signals may include the length and rotation angle of the steering column. The camera calibration device can calculate the extension / retraction length data of the steering column based on the length of the steering column corresponding to the current position of the vehicle-mounted camera and the length of the steering column corresponding to the reference position of the vehicle-mounted camera. Similarly, the deflection angle data of the steering column can be calculated based on the rotation angle of the steering column corresponding to the current position of the vehicle-mounted camera and the rotation angle of the steering column corresponding to the reference position of the vehicle-mounted camera.

[0049] In this embodiment of the disclosure, the mechanical signal of the steering column can be obtained based on an existing positioning device in the vehicle for controlling the vehicle's driving direction. For example, the positioning device for the vehicle's driving direction determines the vehicle's turning radius based on the length of the steering column, controls the actual steering angle of the vehicle based on the rotation angle of the steering column, and the vehicle's control system can jointly determine the vehicle's driving direction based on the vehicle's turning radius and steering angle.

[0050] In other embodiments, the mechanical signals may also directly include data on the extension and retraction length of the steering column and the deflection angle.

[0051] It should be noted that in this embodiment of the present disclosure, the steering column has a rotation center point, and the extension length and deflection angle of the steering column can be determined relative to the rotation center point.

[0052] Figure 2 This is an example diagram illustrating the mechanical information of a steering column according to an embodiment of this disclosure, such as... Figure 2 As shown, 21 marks the steering column, point O is the center of rotation, 22 marks the vehicle-mounted camera at the reference position, and 23 marks the vehicle-mounted camera at the current position. In this disclosure, the extension / retraction length data of the steering column is... Figure 2The ΔL shown is the steering column deflection angle data. Figure 2 Δθ is shown in the figure.

[0053] It is understood that, in this embodiment, since the mechanical signals of the steering column can be obtained, for example, based on the existing positioning device in the vehicle for controlling the direction of travel, without the need for additional complex calculations, the calibration efficiency of the vehicle-mounted camera of this disclosure is improved.

[0054] In some embodiments, determining the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera includes:

[0055] Based on the telescopic length data, the deflection angle data, and the preset angle, a rigid body transformation is performed to obtain the first transformation matrix; wherein, the preset angle is the angle between the lens surface of the vehicle camera and the axis of the steering column.

[0056] In this embodiment, rigid body transformation includes translation and rotation. In this embodiment, considering that the lens surface of the vehicle-mounted camera may not be parallel to the steering column axis during installation (i.e., the vehicle-mounted camera may have a certain elevation or depression angle relative to the steering column axis), the angle of the vehicle-mounted camera's lens surface relative to the steering column axis can be introduced during rigid body transformation to more accurately determine the first transformation matrix, thus adapting to the calibration of vehicle-mounted cameras with different installation methods.

[0057] It is understood that in this embodiment, the preset angle between the lens surface of the vehicle-mounted camera and the axis of the column is used. Since the preset angle is determined when the vehicle-mounted camera is installed, it does not need to be calculated in real time. Therefore, the method of determining the first transformation matrix by rigid body transformation of the telescopic length data, deflection angle data and preset angle can be applied to different vehicle-mounted camera installation methods without affecting efficiency, which can improve the universality of the calibration scheme of this disclosure.

[0058] In some embodiments, the method further includes:

[0059] Acquire a first image captured by the vehicle-mounted camera at the current location, and a second image captured by the vehicle-mounted camera at the reference location;

[0060] After determining the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle camera, feature element recognition and matching are performed on the first image and the second image;

[0061] Based on the matching results of feature elements in the first image and the second image, determine the mapping relationship matrix between the first image and the second image;

[0062] Update the first transformation matrix based on the mapping relationship matrix.

[0063] In this embodiment, after acquiring a first image captured by the vehicle-mounted camera at its current position and a second image captured by the vehicle-mounted camera at a reference position, the camera calibration device performs feature element identification and matching on the first and second images. The feature elements can be specific types of objects or feature points in the images, such as a "B" pillar, a roof boundary point, etc., where the "B" pillar, also known as the central pillar, is located between the front and rear doors of the vehicle. For example, features in each image can be extracted using a neural network model, and matching can be performed based on the features in the first and second images. Subsequently, a mapping relationship matrix between the first and second images is determined based on the positions of features with a matching degree greater than a preset matching threshold in the first and second images.

[0064] It should be noted that, in determining the mapping relationship matrix between the first image and the second image, this disclosure can first preset a transformation matrix, such as a 3*3 zero matrix. The preset transformation matrix is ​​used to map the position (first position) of the feature corresponding to the first image with a matching degree greater than the preset matching threshold. The position (second position) of the feature corresponding to the second image with a matching degree greater than the preset matching threshold is used as the mapping target. The element values ​​in the preset transformation matrix are continuously adjusted until the product of the preset transformation matrix and the first position is the same as that of the second position. The result at this time is the mapping relationship matrix.

[0065] After determining the mapping relationship matrix between the first and second images based on the matching results of feature elements in the first and second images, the first transformation matrix can be updated based on the mapping relationship matrix. In this disclosure, when updating the first transformation matrix based on the mapping relationship matrix, for example, the matrix obtained by weighting and adding the mapping relationship matrix and the first transformation matrix is ​​determined as the updated first transformation matrix. The weights can be preset based on experience; the sum of the weights in the mapping relationship matrix and the first transformation matrix is ​​1, and a larger weight indicates a higher importance of the corresponding matrix.

[0066] It is understandable that after determining the mapping relationship matrix based on visual image information, the first transformation matrix is ​​updated. Since the first transformation matrix is ​​obtained based on mechanical signals, while the mapping relationship matrix is ​​obtained based on visual signals, in this embodiment, errors in mechanical signals, such as those caused by assembly, can be compensated by visual signals, thereby achieving a more accurate determination of the first transformation matrix and improving the calibration accuracy of the vehicle camera.

[0067] In some embodiments, obtaining the first position information of the vehicle-mounted camera's current position relative to a reference position includes:

[0068] Target detection is performed on the first image and the second image respectively to determine the position of the target object in the first image and the position in the second image;

[0069] The first location information is determined to be the deviation of the target object's position in the first image relative to its position in the second image;

[0070] The step of determining the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera includes:

[0071] The first transformation matrix is ​​determined based on the deviation.

[0072] In this embodiment, the first image captured by the vehicle-mounted camera at the current position and the second image captured by the vehicle-mounted camera at the reference position, acquired by the camera calibration device, can be images of the same target object fixed in position within the vehicle cabin. For example, the same target object fixed in position within the vehicle cabin can be a sunroof frame or a "B" pillar, etc.

[0073] This disclosure performs target detection on the first image and the second image respectively. After determining the position of the target object in the first image and the position in the second image, the deviation of the position of the target object in the first image and the second image can be determined, and the deviation of the position can be used as the first position information.

[0074] It should be noted that in this embodiment, the position deviation may be caused by the translation and rotation of the current position of the vehicle camera relative to the reference position. Therefore, the position deviation also includes translation and rotation information. Correspondingly, the first transformation matrix determined based on the position deviation also includes translation and rotation information.

[0075] In this embodiment of the disclosure, when determining the first transformation matrix based on the position deviation, for example, the first transformation matrix of the current position relative to the reference position can be determined by multiplying the position deviation by the camera's intrinsic parameter matrix.

[0076] It is understood that in this embodiment, the detection of target objects in the image information based on vision, and the determination of the first transformation matrix based on the positional deviation of the same target object in different images, that is, the determination of the first transformation matrix using machine vision, is a simple and effective solution.

[0077] In some embodiments, obtaining the third position information of the reference position in the vehicle cabin coordinate system includes:

[0078] Obtain the fourth position information of the rotation center point of the steering column in the vehicle cabin coordinate system;

[0079] Obtain the fifth position information of the reference position relative to the rotation center point of the steering column;

[0080] The third position information of the reference position in the vehicle cabin coordinate system is determined based on the fourth position information and the fifth position information.

[0081] Figure 3 and Figure 4 This is an example diagram showing the three-dimensional data of the steering column's rotation center point in the vehicle cabin coordinate system in an embodiment of this disclosure, as shown below. Figure 3 and Figure 4 As shown, 31 is the origin of the vehicle cabin coordinate system, and 32 is the rotation center of the steering column. The distance Z1 in the height direction, the distance Y1 in the front axle direction, and the distance X1 in the forward direction relative to the origin 31 of the vehicle cabin are the fourth position information. The front axle direction can refer to the direction between the two front wheels when the vehicle is traveling forward; the height direction can refer to the direction perpendicular to the front axle direction, in which the height of the roof above the horizontal ground can be measured; the forward direction can refer to the direction the vehicle is traveling forward, in which the distance between the front and rear seats can be measured.

[0082] also, Figure 2 The position information of the center point of the vehicle-mounted camera 22 at the reference position relative to the rotation center O of the steering column is the fifth position information of the reference position relative to the rotation center point of the steering column. Based on the fourth and fifth position information, this disclosure can determine the third position information of the reference position in the vehicle cabin coordinate system.

[0083] In this embodiment, the third position information of the reference position in the vehicle cabin coordinate system is determined by using the rotation center point of the steering column as an intermediate transformation. In another embodiment, since the reference position is also a pre-determined position, the third position information is also determined directly based on the positional relationship between the reference position in the mechanical design information and the origin of the vehicle cabin coordinate system.

[0084] Figure 5 and Figure 6 This is an example diagram of the three-dimensional data of the reference position in the vehicle cabin coordinate system in an embodiment of this disclosure, such as... Figure 5 and Figure 6 As shown, 31 is the origin of the vehicle cabin coordinate system, and 33 is the reference position. The distance Z2 in the height direction, the distance Y2 in the front axle direction, and the distance X2 in the forward direction of the vehicle cabin relative to the origin 31 of the reference position 33 are the third position information.

[0085] It is understood that, in this embodiment, since the fourth and fifth position information are mechanical design information, they do not require additional complex calculations to obtain, which is beneficial to the calibration efficiency of the vehicle-mounted camera of this disclosure.

[0086] In some embodiments, determining the second transformation matrix of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information includes:

[0087] The translation and rotation matrices of the reference position relative to the vehicle cabin coordinate system are determined based on the third position information and the rotation angle information.

[0088] The translation matrix and the rotation matrix are combined and then multiplied by the first transformation matrix to obtain the second transformation matrix.

[0089] In this embodiment of the disclosure, the translation matrix of the reference position relative to the vehicle cabin coordinate system can be determined based on the third position information. For example, the translation matrix is ​​a matrix composed of X2, Y2, and Z2. Furthermore, when determining the rotation matrix based on the third position information and rotation angle information, three-dimensional spatial modeling can be performed to determine the angles between the line connecting the reference position and the origin of the vehicle cabin coordinate system and each coordinate plane of the vehicle cabin coordinate system. The active rotation matrices of these connecting lines around each axis of the vehicle cabin coordinate system are then obtained. The rotation matrix of this disclosure is obtained by multiplying the active rotation matrices of these connecting lines around each axis.

[0090] As mentioned above, the first transformation matrix includes translation and rotation information of the current position relative to the reference position. After obtaining the translation and rotation matrices of the reference position relative to the vehicle cabin coordinate system, this disclosure can combine the translation and rotation matrices. For example, if the translation matrix is ​​B and the rotation matrix is ​​D, then the matrix [B, D] containing translation and rotation information of the reference position relative to the vehicle cabin coordinate system can be obtained by combining them. After multiplying [B, D] with the first rotation matrix M of the current position relative to the reference position, the second transformation matrix can be obtained. For example, the second transformation matrix is ​​E, where E is [B, D]*M.

[0091] Figure 7 This is a schematic diagram illustrating the principle of a calibration method for a vehicle-mounted camera according to an embodiment of this disclosure, such as... Figure 7As shown, the camera calibration method of this disclosure includes two steps. The first step is to map the real-time position of the vehicle-mounted camera (marked 71) to the reference position of the vehicle-mounted camera (marked 72). During the mapping process, the mechanical extension and rotation signals of the steering column and the interior photos of the vehicle cabin taken by the vehicle-mounted camera can be used. The mechanical extension signal of the steering column includes the extension length data of the steering column mentioned in this disclosure, and the rotation signal includes the deflection angle data mentioned in this disclosure. The interior photos of the vehicle cabin taken by the vehicle-mounted camera include the first image and the second image of this disclosure. The first transformation matrix of this disclosure is obtained through the above mapping. The second step is to map the reference position of the vehicle-mounted camera (marked 72) to the passenger vehicle cabin spatial coordinate system (marked 73). During the mapping process, the mechanical design model of the vehicle cabin can be used. For example, the fourth position information of the rotation center point of the steering column in the cabin coordinate system mentioned in this disclosure, and the fifth position information of the reference position relative to the rotation center point of the steering column can be used to determine the third position information of the reference position in the cabin coordinate system. Based on the rotation angle information of the reference position, the translation matrix and rotation matrix of this disclosure are determined.

[0092] Based on the first transformation matrix, translation matrix, and rotation matrix obtained in the above two steps, the second transformation matrix of the real-time position of the vehicle-mounted camera relative to the vehicle cabin spatial coordinate system in this embodiment can be obtained. It should be noted that since matrix transformation is invertible, if the position of the vehicle-mounted camera relative to the vehicle cabin spatial coordinate system is obtained in advance, the real-time position of the vehicle-mounted camera, that is, the real-time position of the steering column, can be obtained based on the second transformation matrix.

[0093] Figure 8 An example diagram of a camera calibration device provided in an embodiment of this disclosure is shown, consisting of... Figure 8 It can be seen that the camera calibration device 800 includes:

[0094] The first acquisition module 801 is used to acquire first position information of the current position of the vehicle-mounted camera relative to a reference position; wherein, the vehicle-mounted camera is mounted on the steering column;

[0095] The first determining module 802 is used to determine a first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera;

[0096] The second acquisition module 803 is used to acquire the third position information of the reference position in the vehicle cabin coordinate system and the rotation angle information of the steering column in the reference position.

[0097] The second determining module 804 is used to determine the second transformation matrix of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system based on the first transformation matrix, the third position information and the rotation angle information.

[0098] In some embodiments, the first acquisition module 801 is used to read the mechanical signals of the steering column to obtain the extension length data and deflection angle data of the steering column, as the first position information.

[0099] In some embodiments, the first determining module 802 is used to perform rigid body transformation based on the telescopic length data, the deflection angle data, and a preset angle to obtain the first transformation matrix; wherein, the preset angle is the angle between the lens surface of the vehicle camera and the axis of the steering column.

[0100] In some embodiments, the apparatus further includes:

[0101] The third acquisition module is used to acquire a first image captured by the vehicle-mounted camera at the current position, and a second image captured by the vehicle-mounted camera at the reference position;

[0102] The third determining module is used to perform feature element recognition and matching on the first image and the second image after determining the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle camera;

[0103] The fourth determining module is used to determine the mapping relationship matrix between the first image and the second image based on the matching results of feature elements in the first image and the second image;

[0104] An update module is used to update the first transformation matrix based on the mapping relationship matrix.

[0105] In some embodiments, the first acquisition module 801 is configured to perform target detection on the first image and the second image respectively, determine the position of the target object in the first image and the position in the second image; and determine the first position information as the deviation of the position of the target object in the first image relative to its position in the second image;

[0106] The first determining module 802 is used to determine the first transformation matrix based on the deviation.

[0107] In some embodiments, the second acquisition module 802 is used to acquire fourth position information of the rotation center point of the steering column in the vehicle cabin coordinate system; acquire fifth position information of the reference position relative to the rotation center point of the steering column; and determine third position information of the reference position in the vehicle cabin coordinate system based on the fourth position information and the fifth position information.

[0108] In some embodiments, the second determining module 804 is configured to determine the translation matrix and rotation matrix of the reference position relative to the vehicle cabin coordinate system based on the third position information and the rotation angle information; and multiply the translation matrix and the rotation matrix by the first transformation matrix to obtain the second transformation matrix.

[0109] The description of the above apparatus embodiments is similar to that of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the description of the method embodiments of this disclosure for understanding.

[0110] It should be noted that, in the embodiments of this disclosure, if the above-described vehicle camera calibration method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk. Thus, the embodiments of this disclosure are not limited to any specific hardware and software combination.

[0111] Correspondingly, this disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above-described method.

[0112] Correspondingly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0113] Correspondingly, this disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0114] It should be noted that the descriptions of the above-described storage media, computer program products, and device embodiments are similar to the descriptions of the above-described method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the storage media, computer program products, and devices of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0115] It should be noted that, Figure 9 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure, such as... Figure 9 As shown, the hardware entity of the computer device 900 includes: a processor 901, a communication interface 902, and a memory 903, wherein:

[0116] Processor 901 typically controls the overall operation of computer device 900.

[0117] Communication interface 902 enables computer devices to communicate with other terminals or servers over a network.

[0118] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 901 and various modules in the computer device 900. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 901, the communication interface 902, and the memory 903 can be performed via bus 904.

[0119] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above-described embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0121] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0122] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0123] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0124] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0125] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0126] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for calibrating a vehicle camera, characterized by, Applied to a DMS system, the method comprises: obtaining first position information of a current position of a vehicle-mounted camera relative to a reference position; wherein the vehicle-mounted camera is arranged on a steering wheel column; determining a first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera; obtaining a first image collected by the vehicle-mounted camera at the current position and a second image collected by the vehicle-mounted camera at the reference position; performing feature element recognition and matching on the first image and the second image; and determining a mapping relationship matrix of the first image and the second image according to a matching result of feature elements in the first image and the second image; updating the first transformation matrix based on the mapping relationship matrix; obtaining third position information of the reference position in a vehicle cabin coordinate system and rotation angle information of the steering wheel column at the reference position; determining a second transformation matrix of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system according to the first transformation matrix, the third position information and the rotation angle information; detecting a position of a driver in an image frame, and determining a position of the driver relative to the vehicle cabin coordinate system based on the position of the driver, an intrinsic matrix of the vehicle-mounted camera and the second transformation matrix; judging a fatigue state of the driver based on the position of the driver relative to the vehicle cabin coordinate system determined based on continuous multiple image frames; the obtaining of the first position information of the current position of the vehicle-mounted camera relative to the reference position comprises: reading mechanical signals of the steering wheel column to obtain extension length data and deflection angle data of the steering wheel column as the first position information; the mechanical signals are obtained based on a positioning device for controlling a driving direction of the vehicle; the determining of the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera comprises: performing rigid body transformation based on the extension length data, the deflection angle data and a preset angle to obtain the first transformation matrix; wherein the preset angle is an angle of a lens face of the vehicle-mounted camera relative to an axis of the steering wheel column.

2. The method of claim 1, wherein, the obtaining of the first position information of the current position of the vehicle-mounted camera relative to the reference position further comprises: respectively performing target detection on the first image and the second image to determine positions of a target object in the first image and in the second image; determining the first position information as a deviation of the position of the target object in the first image relative to the position in the second image; the determining of the first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle-mounted camera comprises: determining the first transformation matrix based on the deviation.

3. The method according to claim 1 or 2, characterized in that, the obtaining of the third position information of the reference position in the vehicle cabin coordinate system comprises: obtaining fourth position information of a rotation center point of the steering wheel column in the vehicle cabin coordinate system; obtaining fifth position information of the reference position relative to the rotation center point of the steering wheel column; determine third position information of the reference position in a vehicle cabin coordinate system according to the fourth position information and the fifth position information.

4. The method according to claim 1 or 2, characterized in that, The determining, according to the first transformation matrix, the third position information, and the rotation angle information, of a second transformation matrix of the current position of the vehicle camera relative to the vehicle cabin coordinate system comprises: determining a translation matrix and a rotation matrix of the reference position relative to the vehicle cabin coordinate system according to the third position information and the rotation angle information; combining the translation matrix and the rotation matrix and multiplying the combined translation matrix and rotation matrix with the first transformation matrix to obtain the second transformation matrix.

5. A calibration device for a vehicle camera, characterized by The device is applied to a DMS system, and the device comprises: a first obtaining module configured to obtain first position information of a current position of a vehicle camera relative to a reference position, wherein the vehicle camera is arranged on a steering wheel column; a first determining module configured to determine a first transformation matrix of the current position relative to the reference position based on the first position information of the vehicle camera; a third obtaining module configured to obtain a first image captured by the vehicle camera at the current position and a second image captured by the vehicle camera at the reference position; a fourth determining module configured to perform feature element recognition and matching on the first image and the second image, and determine a mapping relationship matrix of the first image and the second image according to a matching result of feature elements in the first image and the second image; an updating module configured to update the first transformation matrix based on the mapping relationship matrix; a second obtaining module configured to obtain third position information of the reference position in a vehicle cabin coordinate system and rotation angle information of the steering wheel column at the reference position; a second determining module configured to determine a second transformation matrix of the current position of the vehicle camera relative to the vehicle cabin coordinate system according to the first transformation matrix, the third position information, and the rotation angle information; The fourth determining module is further configured to detect a position of a driver in an image frame, determine a position of the driver relative to the vehicle cabin coordinate system based on the position of the driver, an intrinsic matrix of the vehicle camera, and the second transformation matrix, and determine a fatigue state of the driver based on the position of the driver relative to the vehicle cabin coordinate system determined based on a plurality of continuous image frames. The first obtaining module is further configured to read a mechanical signal of the steering wheel column to obtain extension length data and deflection angle data of the steering wheel column as the first position information, wherein the mechanical signal is obtained based on a positioning device configured to control a driving direction of the vehicle. The first determining module is further configured to perform rigid body transformation based on the extension length data, the deflection angle data, and a preset angle to obtain the first transformation matrix, wherein the preset angle is an angle of a lens face of the vehicle camera relative to an axis of the steering wheel column.

6. A computer device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 4.

Images