Electronic rearview mirror class ii mirror image correction method, electronic device, and program product

Abstract

This application relates to the field of automotive electronics technology, proposing a method, electronic device, and computer program product for correcting images of a Type II rearview mirror. The method includes: acquiring a homography transformation matrix from the image plane of a real camera in the vehicle's electronic rearview mirror to the image plane of a virtual camera with an ideal viewing posture; wherein the homography transformation matrix is ​​determined based on the parameters of the first camera of the real camera and the second camera of the virtual camera; using the homography transformation matrix, performing perspective transformation processing on the initial Type II mirror image captured by the real camera to obtain a corrected target Type II mirror image. This method, by constructing a virtual camera with an ideal viewing posture in three-dimensional space and reprojecting the Type II mirror image captured by the real camera onto the viewpoint of the virtual camera using the homography transformation matrix, can correct the geometric distortion problem existing in the Type II mirror image, thereby improving the safety and comfort of vehicle driving.

Description

Technical Field

[0001] This application relates to the field of automotive electronics technology, and in particular to a method for correcting images of a Class II rearview mirror, an electronic device, and a computer program product. Background Technology

[0002] The side mirrors of automotive electronic rearview mirrors, also known as Category II mirrors, provide a view of the vehicle's sides and rear. The display quality of the Category II mirror image directly affects the safety and driving experience. In practical applications, to meet mandatory regulatory requirements for field of view, the installation position and orientation of the camera in automotive electronic rearview mirrors are strictly limited. They typically need to be tilted downwards at a certain pitch angle to cover the nearby ground area. Furthermore, the structural tolerances of the vehicle mounting surface and the assembly process inevitably introduce additional attitude deviations such as roll angle deviations. These factors collectively result in geometric distortions in the initial Category II mirror image, including vehicle body tilt, vertical object deformation, and tilted horizon lines, severely impacting driving safety and comfort. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, electronic device, and computer program product for correcting images of a Category II rearview mirror, which can correct geometric distortion problems existing in Category II mirror images, thereby improving the safety and comfort of driving a vehicle.

[0004] The first aspect of this application provides a method for correcting images of a Category II electronic rearview mirror, including: Obtain the homography transformation matrix from the image plane of the real camera of the vehicle electronic rearview mirror to the image plane of the virtual camera with an ideal viewing posture; wherein, the homography transformation matrix is ​​determined according to the parameters of the first camera of the real camera and the parameters of the second camera of the virtual camera; Using a homography transformation matrix, a perspective transformation is performed on the initial Type II mirror image obtained by a real camera to obtain a corrected target Type II mirror image.

[0005] The technical solution of this application pre-constructs a virtual camera with an ideal observation posture in three-dimensional space, that is, the pitch and roll angles of the virtual camera are approximately 0°. Based on the parameters of the first camera of the real camera in the vehicle's electronic rearview mirror and the parameters of the second camera of the virtual camera, a homography transformation matrix is ​​calculated from the image plane of the real camera to the image plane of the virtual camera. For the initial Type II mirror image obtained by the real camera, perspective transformation processing can be performed using this homography transformation matrix to obtain a corrected target Type II mirror image. The above operation uses the homography transformation matrix to reproject the Type II mirror image obtained by the real camera onto the viewpoint of the virtual camera with the ideal posture, which can correct the geometric distortion in the Type II mirror image caused by factors such as pitch and roll angle deviations, generating a geometrically correct Type II mirror image, thereby improving the safety and comfort of vehicle driving.

[0006] In one implementation of this application, the first camera parameters include first Euler angles and the coordinates of the first optical center in the world coordinate system, and the second camera parameters include second Euler angles and the coordinates of the second optical center in the world coordinate system; the homography transformation matrix is ​​determined in the following way: According to the set rotation order around the axis, the first Euler angle is converted into the corresponding first rotation matrix, and the second Euler angle is converted into the corresponding second rotation matrix; The first translation vector is determined based on the first optical center coordinates and the first rotation matrix, and the second translation vector is determined based on the second optical center coordinates and the second rotation matrix; Using the ground plane in the world coordinate system as a reference, the homography transformation matrix is ​​calculated based on the camera intrinsic parameter matrix, the first rotation matrix, the second rotation matrix, the first translation vector, and the second translation vector of the real camera.

[0007] In one implementation of this application, the first Euler angle includes a first pitch angle, a first yaw angle, and a first roll angle, and the second Euler angle includes a second pitch angle, a second yaw angle, and a second roll angle; the second pitch angle and the second roll angle are both 0°, and the second yaw angle is equal to the first yaw angle.

[0008] In one implementation of this application, the origin of the world coordinate system is the ground plane projection point directly below the optical center of the real camera, and the Z-axis is vertically upward; the coordinates of the first optical center are (0, 0, h_orig), where h_orig is the height of the optical center of the real camera above the ground; the coordinates of the second optical center are (0, 0, h_virt), where h_virt is the height of the optical center of the adjustable virtual camera above the ground.

[0009] In one implementation of this application, h_virt = h_orig, or h_virt is adjusted within a set height range via the vehicle-mounted human-machine interface.

[0010] In one implementation of this application, using the ground plane in the world coordinate system as a reference, and based on the camera intrinsic parameter matrix, first rotation matrix, second rotation matrix, first translation vector, and second translation vector of the real camera, a homography transformation matrix is ​​calculated, including: Based on the geometric constraints of imaging the same ground plane by real and virtual cameras, the homography transformation matrix is ​​calculated using the following formula:

[0011] in, H Represents the homography transformation matrix. K This represents the camera intrinsic parameter matrix. Denotes the first rotation matrix. Denotes the second rotation matrix. t orig Denotes the first translation vector. t virt Denotes the second translation vector. n d represents the normal vector of the ground plane, and d represents the distance from the origin of the world coordinate system to the ground plane.

[0012] In one implementation of this application, the ground plane is z=0. n = (0, 0, 1) T , d=0.

[0013] In one implementation of this application, after obtaining the corrected target Type II mirror image, the method further includes: Acquire images of Class VI mirrors from in-vehicle electronic rearview mirrors; The target Class II mirror image and Class VI mirror image are stitched together, and the stitched image is output to a designated area of ​​the vehicle display screen for display.

[0014] A second aspect of this application provides an image correction device for a Category II electronic rearview mirror, comprising: The homography transformation matrix acquisition module is used to acquire the homography transformation matrix from the image plane of the real camera of the vehicle electronic rearview mirror to the image plane of the virtual camera with an ideal observation posture; wherein, the homography transformation matrix is ​​determined according to the first camera parameters of the real camera and the second camera parameters of the virtual camera; The image correction module is used to perform perspective transformation processing on the initial Type II mirror image obtained by a real camera using a homography transformation matrix to obtain the corrected target Type II mirror image.

[0015] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the electronic rearview mirror Class II mirror image correction method as provided in the first aspect of this application.

[0016] A fourth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the electronic rearview mirror Class II mirror image correction method provided in the first aspect of this application.

[0017] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the electronic rearview mirror Class II mirror image correction method provided in the first aspect of this application.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for correcting images of a Category II electronic rearview mirror provided in an embodiment of this application; Figure 2 This is a schematic diagram comparing the real camera pose and the virtual camera pose provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the coordinate system and the geometric relationship between the ground plane involved in the calculation of the homography transformation matrix provided in this application embodiment; Figure 4 This is a schematic diagram of the overall operation process of the electronic rearview mirror Class II mirror image correction method provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the effect obtained by using the electronic rearview mirror Class II mirror image correction method provided in the embodiments of this application; Figure 6 This is a structural framework diagram of an electronic rearview mirror Class II image correction device provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0021] The actual observation posture of the camera in the vehicle electronic rearview mirror is not ideal, with a certain degree of pitch and roll deviation. These factors will cause the following three types of inherent and interrelated geometric distortion problems in the initial Class II mirror image: (1) Vehicle body tilt: Due to the unexpected roll angle of the camera, the outline of the vehicle body in the Class II mirror image will be tilted, which is inconsistent with the driver's upright visual habit and affects the judgment of the relative position of the vehicle body and the lane line; (2) Vertical object deformation: The camera is usually installed with a non-zero pitch angle, which will cause objects that are originally perpendicular to the ground (such as street lights, trees and guardrails) to appear radially tilted in the Class II mirror image, distorting the real spatial relationship; (3) Tilting of the horizon line: The roll angle artificially introduced at the design end to compensate for the vehicle body tilt will also cause the distant horizon line (i.e., the horizon) to be tilted in the Class II mirror image, which will seriously affect the driver's intuitive judgment of the distance and relative speed of vehicles in adjacent lanes. Obviously, the above-mentioned geometric distortion problems will seriously affect the safety and comfort of driving.

[0022] To address the aforementioned technical problems, this application provides a method, electronic device, and computer program product for correcting images of a Category II rearview mirror. By utilizing a homography transformation matrix, the image of a Category II mirror captured by a real camera is reprojected onto the viewpoint of a virtual camera in an ideal posture. This corrects the geometric distortion problem present in the Category II mirror image, thereby improving vehicle driving safety and comfort. For more specific technical implementation details of this application's embodiments, please refer to the method embodiments described below.

[0023] It should be understood that the implementing entity of the various method embodiments proposed in this application can be various types of electronic devices, such as vehicle electronic control units, in-vehicle terminals, controllers of electronic rearview mirrors, mobile phones, tablets, desktop computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0024] Please see Figure 1 This application illustrates a method for correcting images of a Category II electronic rearview mirror, comprising: 101. Obtain the homography transformation matrix from the image plane of the real camera of the vehicle electronic rearview mirror to the image plane of the virtual camera with an ideal observation posture; wherein, the homography transformation matrix is ​​determined according to the parameters of the first camera of the real camera and the parameters of the second camera of the virtual camera; The technical solution of this application utilizes known precise camera parameters to construct a virtual camera with an ideal observation posture in three-dimensional space. That is, the pitch and roll angles of the virtual camera are approximately 0°, with no pitch or roll deviations. In contrast, the real camera of a vehicle's electronic rearview mirror has pitch and roll angles that are not equal to 0°, thus exhibiting certain pitch and roll deviations. The first camera parameters of the real camera can be obtained in advance through a calibration process, mainly including the intrinsic parameter matrix of the side camera. K The parameters include extrinsic Euler angles (including pitch, yaw, and roll angles) and the coordinates of the camera's optical center in the world coordinate system (called optical center coordinates). In practice, the first camera parameters can be derived from the calibration results of the vehicle blind spot monitoring system, such as using the Zhang plane calibration method, where feature points are extracted by placing calibration objects on the ground, and then the above camera parameters are calculated based on the PnP (Perspective-n-Point) method. Since the virtual camera can be regarded as a real camera adjusted to an ideal observation posture, the second camera parameters of the virtual camera can be inferred from the above first camera parameters. For example, the pitch and roll angles in the extrinsic Euler angles can be adjusted to 0°, while other parameters remain unchanged or are reasonably adjusted.

[0025] As an example, Figure 2This is a schematic diagram comparing the real camera pose and the virtual camera pose provided in an embodiment of this application. Figure 2 It can be seen that the observation attitude of a real camera has non-zero pitch angles (Pitch_orig) and roll angles (Roll_orig), which leads to geometric distortion problems in the captured Type II mirror images, such as deformation of vertical objects, vehicle tilt, and tilt of the horizon line. The observation attitude of a virtual camera is ideal, with both pitch angles (Pitch_virt) and roll angles (Roll_virt) equal to 0°. In the captured Type II mirror images, the vehicle body remains vertical, vertical objects are straight, and the horizon line is not tilted, meaning there are no geometric distortion problems.

[0026] Before performing Type II mirror image correction, it is necessary to determine a homography transformation matrix from the image plane of the real camera to the image plane of the virtual camera, based on the parameters of the first camera of the real camera and the second camera of the virtual camera. H The following describes the homography transformation matrix. H The way it is constructed.

[0027] In one implementation of this application, the first camera parameters include first Euler angles and the coordinates of the first optical center in the world coordinate system, and the second camera parameters include second Euler angles and the coordinates of the second optical center in the world coordinate system; the homography transformation matrix is ​​determined in the following way: (1) According to the set rotation order around the axis, convert the first Euler angle into the corresponding first rotation matrix, and convert the second Euler angle into the corresponding second rotation matrix; (2) Determine the first translation vector based on the first optical center coordinates and the first rotation matrix, and determine the second translation vector based on the second optical center coordinates and the second rotation matrix; (3) Using the ground plane in the world coordinate system as the reference, the homography transformation matrix is ​​calculated based on the camera intrinsic parameter matrix, the first rotation matrix, the second rotation matrix, the first translation vector and the second translation vector of the real camera.

[0028] The extrinsic Euler angles of a real camera are called the first Euler angles, and the coordinates of the camera's optical center in the world coordinate system are called the first optical center coordinates C_orig. The extrinsic Euler angles of a virtual camera are called the second Euler angles, and the coordinates of the camera's optical center in the world coordinate system are called the second optical center coordinates C_virt. First, according to the set rotation order around the axes, the first Euler angles (Pitch_orig, Yaw_orig, Roll_orig) are converted into the corresponding 3×3 first rotation matrix R_orig, and the second Euler angles (Pitch_vrit, Yaw_vrit, Roll_vrit) are converted into the corresponding 3×3 second rotation matrix R_virt. Since the use of Euler angles to represent rigid body rotation is related to the rotation order, the rotation order around the axes ZYX needs to be set, i.e., first rotate the yaw angle (Yaw) around the Z-axis, then the pitch angle (Pitch) around the Y-axis, and finally the roll angle (Roll) around the X-axis. Then, the first translation vector T_orig is determined based on the first optical center coordinates C_orig and the first rotation matrix R_orig, and the second translation vector T_virt is determined based on the second optical center coordinates C_virt and the second rotation matrix R_virt. Specifically, T_orig = -R_orig × C_orig and T_virt = -R_virt × C_virt. Finally, using the ground plane in the world coordinate system as a reference, the camera intrinsic parameter matrix of the actual camera is used as the reference. K The homography transformation matrix is ​​calculated using the first rotation matrix R_orig, the second rotation matrix R_virt, the first translation vector T_orig, and the second translation vector T_virt. H .

[0029] In one implementation of this application, the first Euler angle includes a first pitch angle (Pitch_orig), a first yaw angle (Yaw_orig), and a first roll angle (Roll_orig); the second Euler angle includes a second pitch angle (Pitch_vrit), a second yaw angle (Yaw_vrit), and a second roll angle (Roll_vrit); both the second pitch angle (Pitch_vrit) and the second roll angle (Roll_vrit) are 0°, and the second yaw angle (Yaw_vrit) is equal to the first yaw angle (Yaw_orig).

[0030] The virtual camera has an ideal observation attitude. Its second extrinsic Euler angle, Pitch_vrit = 0°, eliminates pitch deviation and ensures the optical axis is parallel to the ground plane. Its second extrinsic Euler angle, Roll_vrit = 0°, eliminates roll deviation and ensures the image coordinate system is vertical. Its second extrinsic Euler angle, Yaw_vrit, maintains an observation direction completely consistent with the real camera. Therefore, we set the second extrinsic Euler angle, Yaw_vrit = Yaw_orig. Thus, the first Euler angle is (Pitch_orig, Yaw_orig, Roll_orig), and the second Euler angle is (0°, Yaw_orig, 0°).

[0031] In one implementation of this application, the origin of the world coordinate system is the ground plane projection point directly below the optical center of the real camera, and the Z-axis is vertically upward; the coordinates of the first optical center are (0, 0, h_orig), where h_orig is the height of the optical center of the real camera above the ground; the coordinates of the second optical center are (0, 0, h_virt), where h_virt is the height of the optical center of the adjustable virtual camera above the ground.

[0032] To simplify calculations, the origin of the world coordinate system can be defined as the ground plane projection point directly below the optical center of the real camera, with the Z-axis pointing vertically upwards. Based on this, the first optical center coordinates C_orig can be represented as (0, 0, h_orig), where h_orig is the ground elevation of the real camera's optical center. The second optical center coordinates C_virt can be represented as (0, 0, h_virt), where h_virt is the ground elevation of the adjustable virtual camera's optical center. Subsequent correction of the Type II mirror image requires perspective transformation. This perspective transformation may result in areas with no effective pixels at the top or bottom of the processed Type II mirror image—areas not covered by the original Type II mirror image. These areas can be eliminated by adjusting the ground elevation h_virt of the virtual camera's optical center.

[0033] In one implementation of this application, h_virt = h_orig, or h_virt is adjusted within a set height range via the vehicle-mounted human-machine interface.

[0034] In practice, a simple setting method is to set h_virt = h_orig. For example, assuming the optical center height (h_orig) of a real camera is 2.2m above the ground, the virtual camera's optical center height (h_virt) can also be set to 2.2m. Another setting method is to allow h_virt to be adjusted within a set height range via the in-vehicle HMI (Human Machine Interface). This allows drivers to easily make personalized adjustments. For example, drivers can adjust the value of h_virt arbitrarily within the height range of [1.8m, 3.2m] via the in-vehicle HMI to suit different drivers' seating posture preferences.

[0035] In one implementation of this application, using the ground plane in the world coordinate system as a reference, and based on the camera intrinsic parameter matrix, first rotation matrix, second rotation matrix, first translation vector, and second translation vector of the real camera, a homography transformation matrix is ​​calculated, including: Based on the geometric constraints of imaging the same ground plane by real and virtual cameras, the homography transformation matrix is ​​calculated using the following formula:

[0036] in, H Represents the homography transformation matrix. K This represents the camera intrinsic parameter matrix. Denotes the first rotation matrix. Denotes the second rotation matrix. t orig Denotes the first translation vector. t virt Denotes the second translation vector. n d represents the normal vector of the ground plane, and d represents the distance from the origin of the world coordinate system to the ground plane.

[0037] In calculating the homography transformation matrix H First, define the ground plane parameters. In the world coordinate system, the ground plane can be represented as... n T ×P_w+d=0, where n Let P_w represent the normal vector to the ground plane, P_w represent any point on the ground plane, and d represent the directed distance from the origin of the world coordinate system to the ground plane. Then, applying the general homography matrix formula, based on multi-view geometry theory, the homography transformation matrix between the images of two cameras observing the same plane is... H It can be represented as: H = K 2( R 2 R 1 T -( R 2 R 1 Tt 1- t 2) n T / ( n T t1+d)) K 1 -1 In the application scenarios of this application embodiment, K 2= K 1= K , R 1= , R 2= , t 1= t orig ，t 2= t virt Substituting these parameters into the general homography matrix formula above, we can deduce the following calculation formula:

[0038] The homography transformation matrix obtained in this way H It includes all the geometric transformation information required to correct the pitch angle (Pitch_orig) and roll angle (Roll_orig). Furthermore, the calculation formula explicitly incorporates ground-plane constraints. n d), which makes it theoretically complete. Even if the origin of the world coordinate system changes, for example, if the origin is defined at the vehicle's center of mass, causing d≠0, the calculation formula still holds. This demonstrates the effectiveness of using this method to calculate the homography transformation matrix. H It has high robustness and versatility.

[0039] In one implementation of this application, the ground plane is z=0. n = (0, 0, 1) T , d=0.

[0040] For simplified calculations, the origin of the world coordinate system is defined as the projection point of the ground plane directly below the optical center of the real camera; that is, the origin lies on the ground plane, and therefore the ground plane can be represented by the equation z=0. Correspondingly, the normal vector of the ground plane is... n = (0, 0, 1) T , set the origin (0, 0, 0) T Substituting P_w into the above ground plane equation n T ×P_w+d=0, which gives d=0. In this case, since... t orig and t virt Only the Z-axis component is present. nLet be the Z-axis vector and d=0. Substituting this into the above calculation formula, the homography transformation matrix can be solved efficiently and easily. H .

[0041] As an example, Figure 3 This is a schematic diagram illustrating the coordinate system and the geometric relationship between the ground plane involved in the homography transformation matrix calculation provided in this application embodiment. (Through...) Figure 3 As can be seen, the origin of the world coordinate system is defined as the ground projection of the optical center of the real camera, and the ground plane equation is constrained as follows: n T Given that ×P_w+d=0, for any physical point P_w on the ground plane, pixel p_orig can be obtained by rotating R_orig and translating T_orig using the extrinsic parameters of the real camera, and then projecting it onto the image plane of the real camera. Pixel p_vrit can be obtained by rotating P_w by R_virt and translating T_virt using the extrinsic parameters of the virtual camera, and then projecting it onto the image plane of the virtual camera. The following mathematical relationship exists between pixel p_orig and pixel p_vrit: p_vrit = H *p_orig. Using this mathematical relationship, the corresponding pixel in the image plane of a virtual camera can be calculated for each pixel in the image plane of the real camera.

[0042] 102. Using the homography transformation matrix, perform perspective transformation on the initial Type II mirror image obtained by a real camera to obtain the corrected target Type II mirror image.

[0043] The homography matrix is ​​obtained by calculating as described above. H Then, using the homography transformation matrix H By performing perspective transformation on the initial Type II mirror image captured by a real camera, the initial Type II mirror image can be reprojected onto the viewpoint of a virtual camera in an ideal pose, thus obtaining the corrected target Type II mirror image. For example, for each frame of the initial Type II mirror image I_orig captured in real time by a real camera, a homography transformation matrix can be used... H By performing perspective transformation on I_orig, a corrected Class II mirror image I_corrected can be generated. I_corrected can be output to a designated area of ​​the vehicle display screen for driver viewing.

[0044] In one implementation of this application, after obtaining the corrected target Type II mirror image, the method further includes: (1) Acquire Class VI mirror images of the vehicle electronic rearview mirror; (2) The target Class II mirror image and Class VI mirror image are stitched together, and the stitched image is output to the designated area of ​​the vehicle display screen for display.

[0045] Category II rearview mirror images provide the vehicle's side and rearward visibility, while Category VI rearview mirror images provide a wide-angle external view. After obtaining the corrected target Category II mirror image, it can be stitched together with the Category VI image from the vehicle's electronic rearview mirror. The stitched image is then output to a designated area of ​​the vehicle's display screen for display. Alternatively, the target Category II mirror image can be merged with the Category IV mirror image to achieve a picture-in-picture display. This setup provides the driver with an image showing a wider external field of vision, thereby further improving driving safety.

[0046] As an example, Figure 4 This is a schematic diagram of the overall operation flow of the electronic rearview mirror Class II mirror image correction method provided in this application embodiment. Figure 4 First, the camera parameters of the actual camera are obtained, including the intrinsic parameter matrix. K The parameters are: pitch angle (Pitch_orig), yaw angle (Yaw_orig), roll angle (Roll_orig), and optical center coordinates (0, 0, h_orig). Then, a virtual camera model is constructed, with camera parameters including: pitch angle (Pitch_vrit = 0°), yaw angle (Yaw_vrit = Yaw_orig), roll angle (Roll_vrit = 0°), and optical center coordinates (0, 0, h_virt). Next, based on the camera parameters of the two cameras, the homography transformation matrix is ​​calculated. H For specific calculation methods, please refer to the relevant descriptions above; then, the homography transformation matrix is ​​used. H By performing a perspective transformation on the initial Type II mirror image, a target Type II mirror image after correcting for geometric distortion can be generated, and finally, the target Type II mirror image is output. Figure 4 The operation process shown utilizes existing camera calibration parameters and innovative three-dimensional geometric transformations to simultaneously solve various geometric distortion problems such as vehicle tilt, vertical object deformation, and tilt of the horizon line, ultimately outputting a Class II mirror image that conforms to human visual habits and reflects realistic spatial relationships.

[0047] The technical solution of this application pre-constructs a virtual camera with an ideal observation posture in three-dimensional space, that is, the pitch and roll angles of the virtual camera are approximately 0°. Based on the parameters of the first camera of the real camera in the vehicle's electronic rearview mirror and the parameters of the second camera of the virtual camera, a homography transformation matrix is ​​calculated from the image plane of the real camera to the image plane of the virtual camera. For the initial Type II mirror image obtained by the real camera, perspective transformation processing can be performed using this homography transformation matrix to obtain a corrected target Type II mirror image. The above operation uses the homography transformation matrix to reproject the initial Type II mirror image obtained by the real camera onto the viewpoint of the ideally postured virtual camera, which can correct the geometric distortion in the Type II mirror image caused by factors such as pitch and roll angle deviations, generating a geometrically correct Type II mirror image, thereby improving the safety and comfort of vehicle driving.

[0048] To facilitate a better understanding of the image correction method for Class II mirrors provided in this application, the following describes the specific image correction principle using the left electronic rearview mirror (CMS Class II mirror) as an example. The image correction principle for the right electronic rearview mirror is exactly the same as that for the left electronic rearview mirror.

[0049] The system software module can be deployed in the CMS all-in-one screen. When the system starts up or the camera calibration is completed, the calibrated left camera parameters can be obtained from the non-volatile memory. These camera parameters mainly include the following necessary information: (1) Intrinsic parameter matrix K

[0050] K Describe the internal optical geometry of the camera, including focal length and principal point.

[0051] (2) Extrinsic parameters in the world coordinate system: These describe the camera's attitude and optical center position in the world coordinate system, defined using Euler angles (pitch_orig, yaw_orig, and roll_orig) and position coordinates. Specifically, pitch_orig = 17.5° (negative values ​​are elevation, positive values ​​are depression, conforming to regulations requiring downward mounting to cover near-ground field of view), yaw_orig = 25° (pointing towards the rear side of the vehicle), and roll_orig = -2.8°. To simplify calculations, the origin of the world coordinate system is defined as the ground projection point directly below the optical center of the actual camera, with the Z-axis perpendicular to the ground and pointing upwards. The optical center coordinates of the actual camera are C_orig = [0, 0, h_orig]. T h_orig=2.2m.

[0052] Furthermore, based on the core idea of ​​this application's embodiments, a virtual camera with an ideal observation posture is constructed. Its camera parameters are set as follows: Pitch_virt = 0° to eliminate pitch deviation and ensure the optical axis is parallel to the ground plane; Yaw_virt = Yaw_orig = 25° to maintain the same observation direction as the real camera; Roll_virt = 0° to eliminate roll deviation and ensure the image coordinate system is vertical; and the virtual camera's optical center coordinates are C_virt = [0, 0, h_virt]. T , h_virt=h_orig=2.2m.

[0053] By rotating around an axis, the Euler angles of the real camera are converted into the rotation matrix R_orig, and the Euler angles of the virtual camera are converted into the rotation matrix Roll_virt. The translation vectors T_orig = -R_orig × C_orig and T_virt = -R_virt × C_virt are calculated. Then, the homography transformation matrix from the real camera image plane to the virtual camera image plane can be calculated according to the formulas described above. H This calculation is based on the geometric constraints of two cameras imaging the same ground plane, and the calculated 3x3 matrix H This includes all the geometric transformation information that can be used to correct pitch and roll deviations.

[0054] During the processing of the real-time video stream, a homography transformation matrix is ​​applied to each initial Type II mirror image I_orig captured by the real camera. H Perspective transformation can be implemented in a practical embedded platform using hardware GDC (Graphics Display Controller), specifically through Python simulation on a PC. The perspective transformation process yields a corrected image of the target Class II mirror, I_corrected, which can then be fed into an image compositing layer and displayed on a dedicated screen of the vehicle's central control screen or electronic rearview mirror.

[0055] Assume homography transformation matrix H as follows: [0.9994, -0.2502, -31.8871] [0.0488, 0.6946, -1.5654] [0.0000, -0.0003, 1.0329] Using this homography transformation matrix HAfter performing perspective transformation on the initial Class II mirror image, the coordinate transformation results of each image corner point are as follows: top corner (0, 0) → (-30.9, -1.5), top right corner (1535, 0) → (1453.3, 71.0), bottom right corner (1535, 1919) → (2021.9, 2781.8), bottom left corner (0, 1919) → (-1014.2, 2637.5).

[0056] The relative rotation matrix R_rel is as follows: [0.9990, -0.0424, -0.0129] [0.0443, 0.9588, 0.2807] [0.0005, -0.2810, 0.9597] The relative translation vector t_rel is [16.9047, 60.0000, 36.2523]. T .

[0057] Please see Figure 5 This is a schematic diagram illustrating the effect obtained by using the electronic rearview mirror Class II mirror image correction method provided in the embodiments of this application. Figure 5 The image on the left is the initial Type II mirror image before correction. Figure 5 The image on the right is the corrected image of the target Type II mirror. (Through...) Figure 5 It is evident that the initial Class II mirror image suffers from geometric distortions such as vehicle tilt, vertical object deformation, and tilted horizon line, while the target Class II mirror image is natural and upright with intuitive spatial relationships, effectively correcting various geometric distortions and facilitating the determination of the position and speed of vehicles approaching from behind.

[0058] In summary, this application provides an image correction method for Category II electronic rearview mirrors based on virtual camera perspective transformation. The image correction effect is comprehensive, simultaneously addressing multiple geometric distortion problems such as vehicle tilt, vertical object deformation, and tilt of the horizon line. Furthermore, this correction method is implemented entirely through software algorithms, requiring no high-precision mechanical adjustments, additional hardware, or complex manual calibration processes, resulting in very low implementation costs. In addition, the algorithm's computational load is clearly defined, enabling real-time processing of high-definition video streams on existing automotive-grade chips, meeting the stringent real-time requirements of in-vehicle systems. Moreover, it is independent of specific scene characteristics, exhibiting strong robustness and good platform portability.

[0059] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0060] The above mainly describes a method for correcting images of a Category II electronic rearview mirror. The following will describe a device for correcting images of a Category II electronic rearview mirror.

[0061] Please see Figure 6 One embodiment of an image correction device for a Class II electronic rearview mirror in this application includes: The homography transformation matrix acquisition module 601 is used to acquire the homography transformation matrix from the image plane of the real camera of the vehicle electronic rearview mirror to the image plane of the virtual camera with an ideal observation posture; wherein, the homography transformation matrix is ​​determined according to the first camera parameters of the real camera and the second camera parameters of the virtual camera; The image correction module 602 is used to perform perspective transformation processing on the initial Type II mirror image obtained by a real camera using a homography transformation matrix to obtain the corrected target Type II mirror image.

[0062] In one implementation of this application, the first camera parameters include first Euler angles and first optical center coordinates in the world coordinate system, and the second camera parameters include second Euler angles and second optical center coordinates in the world coordinate system; the electronic rearview mirror Class II mirror image correction device further includes: The Euler angle conversion module is used to convert the first Euler angle into the corresponding first rotation matrix and the second Euler angle into the corresponding second rotation matrix according to the set rotation sequence around the axis. The translation vector determination module is used to determine a first translation vector based on the first optical center coordinates, and to determine a second translation vector based on the second optical center coordinates; The homography transformation matrix calculation module is used to calculate the homography transformation matrix with the ground plane in the world coordinate system as the reference reference, based on the camera intrinsic parameter matrix, the first rotation matrix, the second rotation matrix, the first translation vector, and the second translation vector of the real camera.

[0063] In one implementation of this application, the first Euler angle includes a first pitch angle, a first yaw angle, and a first roll angle, and the second Euler angle includes a second pitch angle, a second yaw angle, and a second roll angle; the second pitch angle and the second roll angle are both 0°, and the second yaw angle is equal to the first yaw angle.

[0064] In one implementation of this application, the origin of the world coordinate system is the ground plane projection point directly below the optical center of the real camera, and the Z-axis is vertically upward; the coordinates of the first optical center are (0, 0, h_orig), where h_orig is the height of the optical center of the real camera above the ground; the coordinates of the second optical center are (0, 0, h_virt), where h_virt is the height of the optical center of the adjustable virtual camera above the ground.

[0065] In one implementation of this application, h_virt = h_orig, or h_virt is adjusted within a set height range via the vehicle-mounted human-machine interface.

[0066] In one implementation of this application, the homography transformation matrix calculation module includes: The matrix calculation unit is used to calculate the homography transformation matrix based on the geometric constraints of imaging the same ground plane by both real and virtual cameras, using the following formula:

[0067] in, H Represents the homography transformation matrix. K This represents the camera intrinsic parameter matrix. Denotes the first rotation matrix. Denotes the second rotation matrix. t orig Denotes the first translation vector. t virt Denotes the second translation vector. n d represents the normal vector of the ground plane, and d represents the distance from the origin of the world coordinate system to the ground plane.

[0068] In one implementation of this application, the ground plane is z=0. n = (0, 0, 1) T , d=0.

[0069] In one implementation of this application, the electronic rearview mirror Category II mirror image correction device further includes: The Class VI mirror image acquisition module is used to acquire Class VI mirror images of the vehicle electronic rearview mirror; The image display module is used to stitch together the target Class II mirror image and Class VI mirror image, and output the stitched image to a designated area of ​​the vehicle display screen for display.

[0070] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the electronic rearview mirror Class II mirror image correction method as described in any of the above embodiments.

[0071] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the electronic rearview mirror Class II mirror image correction method as described in any of the above embodiments.

[0072] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 7As shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, it implements the steps in the embodiments of the various electronic rearview mirror Type II mirror image correction methods described above, for example... Figure 1 Steps 101 to 102 are shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of modules 601 to 602 are shown.

[0073] The computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in the electronic device 7.

[0074] The processor 70 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0075] The memory 71 can be an internal storage unit of the electronic device 7, such as a hard disk or memory. The memory 71 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 71 can include both internal and external storage units of the electronic device 7. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0084] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for correcting images of a Class II electronic rearview mirror, characterized in that, include: Obtain the homography transformation matrix from the image plane of the real camera of the vehicle electronic rearview mirror to the image plane of the virtual camera with an ideal viewing posture; wherein, the homography transformation matrix is ​​determined according to the first camera parameters of the real camera and the second camera parameters of the virtual camera; Using the homography transformation matrix, the initial Type II mirror image obtained by the real camera is subjected to perspective transformation processing to obtain the corrected target Type II mirror image.

2. The method as described in claim 1, characterized in that, The first camera parameters include first Euler angles and first optical center coordinates in the world coordinate system; the second camera parameters include second Euler angles and second optical center coordinates in the world coordinate system; the homography transformation matrix is ​​determined in the following way: According to the set rotation sequence around the axis, the first Euler angle is converted into the corresponding first rotation matrix, and the second Euler angle is converted into the corresponding second rotation matrix; A first translation vector is determined based on the first optical center coordinates and the first rotation matrix, and a second translation vector is determined based on the second optical center coordinates and the second rotation matrix; Using the ground plane in the world coordinate system as a reference, the homography transformation matrix is ​​calculated based on the camera intrinsic parameter matrix of the real camera, the first rotation matrix, the second rotation matrix, the first translation vector, and the second translation vector.

3. The method as described in claim 2, characterized in that, The first Euler angle includes a first pitch angle, a first yaw angle, and a first roll angle; the second Euler angle includes a second pitch angle, a second yaw angle, and a second roll angle; the second pitch angle and the second roll angle are both 0°, and the second yaw angle is equal to the first yaw angle.

4. The method as described in claim 2, characterized in that, The origin of the world coordinate system is the ground plane projection point directly below the optical center of the real camera, with the Z-axis pointing vertically upwards; the first optical center coordinates are (0, 0, h_orig), where h_orig is the ground height of the optical center of the real camera; the second optical center coordinates are (0, 0, h_virt), where h_virt is the adjustable ground height of the optical center of the virtual camera.

5. The method as described in claim 4, characterized in that, Set h_virt=h_orig, or set h_virt to be adjusted within a set height range via the vehicle's human-machine interface.

6. The method as described in claim 2, characterized in that, The homography transformation matrix is ​​calculated using the ground plane in the world coordinate system as a reference, based on the camera intrinsic parameter matrix of the real camera, the first rotation matrix, the second rotation matrix, the first translation vector, and the second translation vector. This includes: Based on the geometric constraints of the imaging of the same ground plane by the real camera and the virtual camera, the homography transformation matrix is ​​calculated using the following formula: in, H This represents the homography transformation matrix. K This represents the intrinsic parameter matrix of the camera. Denotes the first rotation matrix. Denotes the second rotation matrix. t orig Represents the first translation vector. t virt This represents the second translation vector. n The vector represents the normal vector of the ground plane, and d represents the distance from the origin of the world coordinate system to the ground plane.

7. The method as described in claim 6, characterized in that, The ground plane is z=0. n = (0, 0, 1) T , d=0.

8. The method according to any one of claims 1 to 7, characterized in that, After obtaining the corrected target Type II mirror image, the method further includes: Acquire the Class VI mirror image of the vehicle electronic rearview mirror; The target Class II mirror image and the Class VI mirror image are stitched together, and the stitched image is output to a designated area of ​​the vehicle display screen for display.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the electronic rearview mirror Class II mirror image correction method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, When the computer program product is run on an electronic device, the electronic device performs the electronic rearview mirror Class II mirror image correction method as described in any one of claims 1 to 8.

Images