Calibration parameter acquisition method, image processing method and device, and electronic device

Abstract

The application discloses a kind of calibration parameter acquisition method, image processing method and its device, electronic equipment. Among them, the calibration parameter method includes: obtaining the first image containing first calibration pattern;Display the second image containing second calibration pattern;Obtain the third image containing the first calibration pattern and the second calibration pattern;According to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern and the second calibration pattern contained in the third image, determine the transformation relationship of the first image and the second image.The present application can solve the technical problem that the prior art cannot obtain calibration parameters using a simple method in an assisted driving system, and then convert real-time external images into virtual images to assist in displaying line-of-sight blind areas.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology, and in particular, to a kind of calibration parameter acquisition method, image processing method and its device, electronic equipment. BACKGROUND

[0002] With the rise of the tide of "intelligent" of car, computer vision, artificial intelligence, virtual \ augmented reality and other technologies are widely used in the driving and safety field of car. Since driver in the driving process will exist sight blind area due to the obstruction of some parts (for example, A column) of car, it is easy to cause safety hazard. The A column of car refers to the column between engine compartment and driver's cabin, above left and right rearview mirrors, left and right front of vehicle connects roof and front compartment. In terms of vehicle structure, A column strength needs to meet roof crush standard, and needs to protect passengers as much as possible when vehicle is hit or rolls over. Therefore, in addition to using the material with the highest strength of the whole vehicle, A column also needs to ensure relatively large cross-sectional area to improve safety level. However, such large cross-sectional area non-transparent area is easy to block the driver's view, and most of the current auxiliary driving systems cannot display the blocked sight blind area, so as to reduce traffic accidents caused by the existence of sight blind area.

[0003] Therefore, it is necessary to propose an image processing technology, which can convert the blocked blind area image into a virtual image displayed on the display device inside the vehicle to assist the driver to drive safely and reduce traffic accidents. SUMMARY

[0004] The embodiment of the present application provides a kind of calibration parameter acquisition method, image processing method and its device, electronic equipment, to at least solve the technical problem in prior art that auxiliary driving system cannot obtain calibration parameter by simple method, and then convert real-time external image into virtual image to assist display sight blind area.

[0005] According to an aspect of the embodiment of the present application, a calibration parameter acquisition method is provided, comprising: obtaining a first image containing a first calibration pattern;Display a second image containing a second calibration pattern;Obtain a third image containing the first calibration pattern and the second calibration pattern;According to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern and the second calibration pattern contained in the third image, determine the transformation relationship of the first image and the second image.

[0006] Optionally, the determining the transformation relationship between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the third image and the second calibration pattern contained in the third image comprises: determining a first transformation relationship between the first calibration pattern in the first image and the first calibration pattern in the third image; determining a second transformation relationship between the second calibration pattern in the third image and the second calibration pattern in the second image; and determining the transformation relationship between the first image and the second image according to the first transformation relationship and the second transformation relationship.

[0007] Optionally, the first image containing the first calibration pattern is acquired by using a first camera outside the vehicle.

[0008] Optionally, the first camera is mounted on a rearview mirror of the vehicle and faces a side front of the vehicle.

[0009] Optionally, the first image is an all-coverage image acquired from outside the vehicle.

[0010] Optionally, the first calibration pattern is located on a calibration object at a side front of the vehicle head, and the first calibration pattern contains N feature points, N being an integer not less than 3.

[0011] Optionally, after the first image containing the first calibration pattern is acquired, the calibration parameter acquisition method further comprises a de-distortion step.

[0012] Optionally, the second image containing the second calibration pattern is displayed by using a display device inside the vehicle.

[0013] Optionally, the display device inside the vehicle is mounted on a vehicle A-pillar and displays towards a driver.

[0014] Optionally, the third image containing the first calibration pattern and the second calibration pattern is acquired by using a second camera inside the vehicle.

[0015] Optionally, the second camera is a camera simulating a human eye perspective.

[0016] Optionally, the third image is an image with a blind area acquired from inside the vehicle.

[0017] Optionally, the first calibration pattern and the second calibration pattern contain not less than 3 identifiable feature points which are not collinear at different times.

[0018] Optionally, the style of the identifiable feature points is at least one of the following: a two-dimensional code, an array of dots, a chessboard.

[0019] Optionally, the method further comprises: adjusting the second camera inside the vehicle to M preset positions which are not collinear for multiple times, and obtaining M third images containing the first calibration pattern and the second calibration pattern, wherein M is an integer not less than 3; and determining the transformation relationship between the M sets of the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern and the second calibration pattern contained in the M third images.

[0020] Optionally, the M preset positions are M positions in a normal eye position change range of the driver during driving.

[0021] Optionally, the method further comprises: obtaining the projection position of the optical center of the second camera inside the vehicle on the third camera using the third camera.

[0022] According to another aspect of the embodiments of the present application, an image processing method is provided, comprising: obtaining a real-time position of an eye inside a cab; obtaining a real-time external image; determining a real-time transformation relationship between the real-time external image and a virtual image displayed on a display device inside the vehicle according to the real-time position of the eye and pre-obtained calibration parameters, wherein the pre-obtained calibration parameters are obtained using any of the above calibration parameter obtaining methods; and transforming the real-time external image into the virtual image according to the real-time transformation relationship.

[0023] Optionally, the real-time external image is obtained using a first camera outside the vehicle.

[0024] Optionally, the real-time position of the eye inside the cab is obtained using a third camera inside the vehicle.

[0025] According to another aspect of the embodiments of the present application, a calibration parameter obtaining device is provided, comprising: a first camera unit configured to obtain a first image containing a first calibration pattern; a display unit configured to display a second image containing a second calibration pattern; a third camera unit configured to obtain a third image containing the first calibration pattern and the second calibration pattern; and a parameter determining unit configured to determine a transformation relationship between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the third image and the second calibration pattern.

[0026] Optionally, the parameter determining unit comprises: a first determining sub-unit configured to determine a first transformation relationship between the first calibration pattern in the first image and the first calibration pattern in the third image; a second determining sub-unit configured to determine a second transformation relationship between the second calibration pattern in the third image and the second calibration pattern in the second image; and a third determining sub-unit configured to determine the transformation relationship between the first image and the second image according to the first transformation relationship and the second transformation relationship.

[0027] Optionally, the first camera unit is mounted outside the vehicle.

[0028] Optionally, the first camera unit is mounted on a rearview mirror of the vehicle, and faces a side front of the vehicle.

[0029] Optionally, the first image is an all-coverage image obtained from outside the vehicle.

[0030] Optionally, the first calibration pattern is located on a calibration object in a side front of the vehicle, and the first calibration pattern comprises N feature points, N being an integer not less than 3.

[0031] Optionally, the N feature points of the first calibration pattern are completely covered by a field of view of the first camera unit.

[0032] Optionally, the calibration parameter obtaining device further comprises a processing unit configured to perform distortion correction on the obtained first image containing the first calibration pattern.

[0033] Optionally, the display device is mounted inside the vehicle.

[0034] Optionally, the display device is mounted on an A-pillar of the vehicle and faces the driver.

[0035] Optionally, the second camera unit is mounted inside the vehicle.

[0036] Optionally, the second camera unit is a camera simulating a human eye perspective.

[0037] Optionally, the third image is an image with a blind area obtained from inside the vehicle.

[0038] Optionally, the first calibration pattern and the second calibration pattern comprise at least 3 identifiable feature points which are not collinear.

[0039] Optionally, the identifiable feature points are in a pattern of at least one of the following: a two-dimensional code, an array of dots, a checkerboard.

[0040] Optionally, the calibration parameter obtaining device further comprises adjusting the second camera unit inside the vehicle to M preset positions which are not collinear, obtaining M third images containing the first calibration pattern and the second calibration pattern, and determining M sets of transformation relationships between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern and the second calibration pattern contained in the M third images, M being an integer not less than 3.

[0041] Optionally, the M preset positions which are not collinear are M positions in a normal range of positions of the driver's eyes during driving.

[0042] Optionally, the calibration parameter acquisition device further comprises: using the third camera unit to acquire a projection position of an optical center of the second camera unit on the third camera unit.

[0043] According to another aspect of the embodiments of the present application, a storage medium is also provided, comprising a stored program, wherein the storage medium controls a device in which the storage medium is located to execute the calibration parameter acquisition method according to any one of the preceding aspects when the program is executed.

[0044] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the calibration parameter acquisition method according to any one of the preceding aspects by executing the executable instructions.

[0045] In the calibration parameter acquisition method according to the embodiments of the present application, by performing the following steps: acquiring a first image containing a first calibration pattern; displaying a second image containing a second calibration pattern; acquiring a third image containing the first calibration pattern and the second calibration pattern; determining a transformation relationship between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern and the second calibration pattern contained in the third image. The technical problem that the prior art auxiliary driving system cannot obtain calibration parameters by a simple method, and then convert real-time external images into virtual images to assist in displaying the line-of-sight blind area is solved.

[0046] The present application proposes a simple calibration parameter acquisition method, which does not require real-time three-dimensional rendering, can significantly reduce the requirement for computing power, and based on the pre-acquired calibration parameters, converts the real-time external images of the blind area position that is blocked into virtual images displayed on the display device in the vehicle interior, and by obtaining the real-time position of the human eye, the virtual images can be moved in real time following the driver's line of sight, and can be aligned with the real-time external images, so that the display screen picture in the driver's field of view at the current time and the vehicle exterior environment are aligned, achieving an "immersive" augmented reality experience. The image processing method only needs two camera devices and one display device, does not require a three-dimensional reconstruction process, and does not require complex camera internal and external parameter calibration, the real-time performance depends on the lowest frame rate of the two cameras and the display device, and the requirements for balancing computing power, cost and user experience can be met. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0048] Figure 1 is a flow chart of an optional calibration parameter acquisition method according to an embodiment of the present application;

[0049] Figure 2 is a flow chart of an optional image processing method according to an embodiment of the present application;

[0050] Figure 3 is a structural diagram of an optional calibration parameter acquisition device according to an embodiment of the present application;

[0051] Figure 4 is a structural diagram of an optional image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0053] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the order used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. Although a logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that herein.

[0054] Reference Figure 1 is a flow chart of an optional calibration parameter acquisition method according to an embodiment of the present application. As shown in Figure 1 the calibration parameter acquisition method includes the following steps:

[0055] S100, acquiring a first image containing a first calibration pattern;

[0056] In an alternative embodiment, the first image containing the first calibration pattern can be acquired by using a first camera outside the vehicle, which can be a camera mounted on a rearview mirror (including a left rearview mirror and / or a right rearview mirror) of the vehicle, facing the side front (including the left front and / or the right front) of the vehicle. The first image is an image acquired from outside the vehicle without blind area. The first calibration pattern is located on a calibration object in the side front of the vehicle head, and the first calibration pattern contains N feature points, which can be distributed on the first calibration pattern in a whole or separated manner, and N is an integer not less than 3.

[0057] In an alternative embodiment, after acquiring the first image containing the first calibration pattern by using the first camera outside the vehicle, a step of removing distortion can be further included, for example, using Zhang's calibration method to obtain distortion parameters, and correcting the distortion of the first image based on the distortion parameters. Removing the distortion of the first image can improve the accuracy of the calibration parameters and improve the image alignment effect achieved based on the calibration parameters.

[0058] Generally, the calibration environment is in an indoor garage or on a car production line, and the calibration object can be vertically placed in the left front and / or right front of the vehicle head, so that the N feature points of the first calibration pattern on the calibration object are completely covered by the field of view of the first camera. In a specific embodiment, the calibration object can be vertically placed about 3-4 meters away from the side front (at least one of the left front or the right front) of the vehicle head. The normal vector of the plane of the calibration object is parallel to the optical axis direction of the first camera outside the vehicle. The calibration object has a first calibration pattern containing N feature points, and when the driver sits on the driver's seat and observes the calibration object, the N feature points of the first calibration pattern are uniformly distributed on both sides of the vehicle A pillar. In a specific embodiment, at least 4 feature points of the first calibration pattern are located at the four corners of the calibration board, and when the driver sits on the driver's seat and observes the calibration board, the at least 4 feature points of the first calibration pattern are uniformly distributed on both sides of the vehicle A pillar and are not blocked by the vehicle structure itself.

[0059] S102, displaying a second image containing a second calibration pattern.

[0060] In an alternative embodiment, the second image containing the second calibration pattern can be displayed by using a display device inside the vehicle. In a specific embodiment, the display device inside the vehicle is mounted on the vehicle A pillar and faces the driver display.

[0061] S104: acquiring a third image containing the first calibration pattern and the second calibration pattern;

[0062] In an optional embodiment, a third image containing the first calibration pattern and the second calibration pattern can be acquired using a second camera inside the vehicle, which is a camera simulating the perspective of a human eye, to assist calibration. The third image is a blind area image acquired from inside the vehicle, for example, a blind area image acquired by a camera simulating the perspective of a human eye.

[0063] S106: Determine the transformation relationship between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the third image, and the second calibration pattern contained in the third image.

[0064] In an optional embodiment, step S106 includes:

[0065] S1060: Determine a first transformation relationship between the first calibration pattern in the first image and the first calibration pattern in the third image.

[0066] S1062: Determine a second transformation relationship between the second calibration pattern in the third image and the second calibration pattern in the second image.

[0067] S1064: Determine the transformation relationship between the first image and the second image according to the first transformation relationship and the second transformation relationship.

[0068] It should be noted that the first calibration pattern and the second calibration pattern contain no less than three different collinear identifiable feature points, which can be a two-dimensional code, a circular dot array, a checkerboard, etc., and are not limited to any other pattern that can provide image features. If the feature points are in the form of a two-dimensional code, they can not be laid out in a large area on the calibration pattern, and if they are in the form of a circular dot array, they are more robust and can avoid the influence of reflection of the display device. In a specific embodiment, the feature point pattern of the first calibration pattern is a two-dimensional code, and the feature point pattern of the second calibration pattern is a circular dot array. The resolution of the second calibration pattern is consistent with the resolution of the A-pillar display device, which can reduce the scaling step that needs to be additionally performed due to inconsistent resolutions in the calibration parameter acquisition method.

[0069] The calibration parameter acquisition method provided by the embodiments of the present application does not require real-time three-dimensional rendering, and can significantly reduce the requirement for computing power.

[0070] Since the driver's eye position changes due to turning head, lowering head, raising head and other actions during driving, in order to better simulate the human eye view angle, the calibration parameter acquisition method provided in the above embodiment can further include: adjusting the second camera inside the vehicle to M preset positions which are not collinear for multiple times, and acquiring M third images containing the first calibration pattern and the second calibration pattern, so as to determine the transformation relationship of M groups of first images and second images according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, and the first calibration pattern and the second calibration pattern contained in the M third images. The M preset positions which are not collinear are M positions in the normal change range of the driver's eye position during driving, and M is an integer greater than or equal to 3.

[0071] According to another optional calibration parameter acquisition method of the embodiment of the application, in addition to the steps S100-S106, the method can further include a step S108 of acquiring the projection position of the optical center of the second camera inside the vehicle on the third camera by using the third camera. In an optional embodiment, the third camera can be a driver monitoring system (DMS) camera. The driver monitoring system (DMS) is used to detect the driver's behavior by a visual method, such as closing eyes, blinking, gazing direction, head movement, etc. The DMS camera can be installed at the bottom of the A-pillar and faces the driver's face.

[0072] Reference Figure 2 An optional flowchart of an image processing method according to an embodiment of the application is provided. As shown in FIG. 8, the image processing method includes the following steps: Figure 4

[0073] S200, acquiring the real-time position of the driver's eye in the cab;

[0074] In an optional embodiment, the real-time position of the driver's eye in the cab can be acquired by using a third camera inside the vehicle. The third camera can be a driver monitoring system (DMS) camera.

[0075] S202, acquiring a real-time external image;

[0076] In an optional embodiment, the real-time external image can be acquired by using a first camera outside the vehicle. The first camera can be a camera installed on the rearview mirror of the vehicle (including the left rearview mirror and / or the right rearview mirror) and faces the front side of the vehicle (including the left front side and / or the right front side).

[0077] S204, determining the real-time transformation relationship between the real-time external image and the virtual image displayed on the display device inside the vehicle according to the real-time position of the driver's eye and the pre-acquired calibration parameter; ​

[0078] In an alternative embodiment, the pre-acquired calibration parameters are obtained according to the calibration parameter acquisition method described above.

[0079] S206: Transform the real-time external image into a virtual image according to the real-time transformation relationship.

[0080] The image processing method provided by the embodiment of the present application can transform the real-time external image of the blind area position that is blocked into a virtual image displayed on the display device inside the vehicle based on the pre-acquired calibration parameters, and by obtaining the real-time position of the human eye, the virtual image can be real-time moved following the driver's line of sight while being aligned with the real-time external image, so that the display screen picture in the driver's field of view at the current time and the external environment of the vehicle are aligned, achieving an "immersive" augmented reality experience. The image processing method only needs two cameras and one display device to achieve, without the process of three-dimensional reconstruction and the need for complex camera internal and external parameter calibration. The real-time performance depends on the lowest frame rate among the two cameras and the display device, which can meet the demand of balancing computing power, cost and user experience.

[0081] The device position involved in the above-mentioned calibration parameter acquisition method and image processing method is described in the form of a schematic according to the transformation of the real-time external image of the A-pillar blind area into a virtual image displayed on the display device inside the A-pillar of the vehicle. Since the front end of the vehicle is provided with a rearview mirror, a front bumper, a sun visor, an engine cover, the rear end is provided with a rear baffle, the left side is provided with a left guardrail, the right side is provided with a right guardrail and other blocking components, those skilled in the art can know that by reasonably transforming the above-mentioned calibration parameter acquisition method and image processing method, the real-time external image of the blind area position corresponding to the above-mentioned blocking components of the vehicle can be transformed into a virtual image displayed on the display device inside the vehicle, and other blind area positions can be, for example, a rearview mirror blind area, a front end blind area, a rear end blind area, a left side blind area, a right side blind area, etc.

[0082] In addition, the above-mentioned calibration parameter acquisition method and image processing method provided by the embodiment of the present application are not limited to ordinary passenger vehicles, but can also be applied to other scenarios, such as dump trucks, cement trucks, trucks, buses, ships, airplanes, etc.

[0083] According to another aspect of the embodiment of the present application, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the calibration parameter acquisition method or the image processing method of any one of the above-mentioned methods by executing the executable instructions.

[0084] According to another aspect of the embodiment of the present application, a storage medium is also provided, which comprises a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the calibration parameter acquisition method or the image processing method of any one of the above-mentioned methods.

[0085] Reference Figure 3 is a structural block diagram of an optional calibration parameter acquisition device according to an embodiment of the present application. The calibration parameter acquisition device 30 comprises:

[0086] a first camera unit 300, configured to acquire a first image containing a first calibration pattern;

[0087] In an optional embodiment, the first camera unit is installed outside the vehicle. For example, the first camera unit can be a camera installed on the rearview mirror of the vehicle (including the left rearview mirror and / or the right rearview mirror), facing the side front of the vehicle (including the left front and / or the right front). The first image is an image acquired from outside the vehicle without blind area. The first calibration pattern is located on a calibration object in the side front of the vehicle head, and the first calibration pattern contains N feature points, which can be distributed on the first calibration pattern in a whole or separated manner, and N is an integer not less than 3.

[0088] In an optional embodiment, after acquiring the first image containing the first calibration pattern using the first camera unit outside the vehicle, a step of removing distortion can be further included, for example, the first camera unit uses Zhang's calibration method to obtain distortion parameters, and corrects the distortion of the first image based on the distortion parameters. Removing the distortion of the first image can improve the accuracy of the calibration parameters and improve the image alignment effect achieved based on the calibration parameters.

[0089] In an optional embodiment, the calibration parameter acquisition device 30 further comprises a processing unit 301, configured to remove the distortion of the acquired first image containing the first calibration pattern, for example, using Zhang's calibration method to obtain distortion parameters, and correcting the distortion of the first image based on the distortion parameters. Removing the distortion of the first image can improve the accuracy of the calibration parameters and improve the image alignment effect achieved based on the calibration parameters.

[0090] Generally, the calibration environment is in an indoor garage or a car production line, and the calibration object can be vertically placed in the left front and / or right front of the vehicle head, so that the N feature points of the first calibration pattern on the calibration object are completely covered by the field of view of the first camera unit. In a specific embodiment, the calibration object can be vertically placed about 3 to 4 meters away from the side front (at least one of the left front or the right front) of the vehicle head. The normal vector of the plane of the calibration object is parallel to the optical axis direction of the first camera unit outside the vehicle. The calibration object has a first calibration pattern containing N feature points, and when the driver sits on the driver's seat and observes the calibration object, the N feature points of the first calibration pattern are uniformly distributed on both sides of the vehicle A pillar. In a specific embodiment, at least 4 feature points of the first calibration pattern are respectively located at the four corners of the calibration board, and when the driver sits on the driver's seat and observes the calibration board, the at least 4 feature points of the first calibration pattern are uniformly distributed on both sides of the vehicle A pillar and are not blocked by the vehicle itself.

[0091] The display unit 302 is configured to display a second image containing a second calibration pattern.

[0092] In an optional embodiment, the display unit is installed inside the vehicle. For example, the display device can be installed on the A-pillar of the vehicle and faces the driver.

[0093] The third camera unit 304 is configured to acquire a third image containing the first calibration pattern and the second calibration pattern.

[0094] In an optional embodiment, the second camera unit is installed inside the vehicle. For example, the second camera unit can be a camera simulating the perspective of human eyes to assist calibration. The third image is a blind area image acquired from inside the vehicle, for example, a blind area image acquired by a camera simulating the perspective of human eyes.

[0095] The parameter determination unit 306 is configured to determine the transformation relationship between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the third image, and the second calibration pattern contained in the third image.

[0096] In an optional embodiment, the parameter determination unit 306 comprises:

[0097] The first determination sub-unit 3060 is configured to determine a first transformation relationship between the first calibration pattern in the first image and the first calibration pattern in the third image.

[0098] The second determination sub-unit 3062 is configured to determine a second transformation relationship between the second calibration pattern in the third image and the second calibration pattern in the second image.

[0099] The third determination sub-unit 3064 is configured to determine the transformation relationship between the first image and the second image according to the first transformation relationship and the second transformation relationship.

[0100] It should be noted that the first calibration pattern and the second calibration pattern contain at least 3 identifiable feature points that are not collinear. The feature points can be in the form of a two-dimensional code, an array of dots, a checkerboard, or any other pattern that can provide image features. If the feature points are in the form of a two-dimensional code, they can not be laid out in a large area on the calibration pattern. If the feature points are in the form of an array of dots, they are more robust and can avoid the influence of reflection from the display device. In a specific embodiment, the feature points of the first calibration pattern are in the form of a two-dimensional code, and the feature points of the second calibration pattern are in the form of an array of dots. The resolution of the second calibration pattern is consistent with the resolution of the A-pillar display device, which can reduce the need for an additional scaling step in the calibration parameter acquisition method due to inconsistent resolutions.

[0101] The calibration parameter acquisition device provided by the embodiment of the present application does not need real-time three-dimensional rendering, and can significantly reduce the requirement for computing power.

[0102] In order to better simulate the human eye perspective, the calibration parameter acquisition device provided by the above embodiment can further include: adjusting the second camera unit inside the vehicle to M preset positions which are not collinear for multiple times, and acquiring M third images containing the first calibration pattern and the second calibration pattern, so as to determine the transformation relationship of the M groups of first images and second images according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, and the first calibration pattern and the second calibration pattern contained in the M third images. The M preset positions which are not collinear are M positions in the normal change range of the human eye position of the driver during driving, and M is an integer greater than or equal to 3.

[0103] According to another optional calibration parameter acquisition device of the embodiment of the present application, in addition to the above-mentioned units 300-306, the device can further include: using the third camera unit to acquire the projection position of the optical center of the second camera unit inside the vehicle on the third camera unit. In an optional embodiment, the third camera unit can be a driver monitoring system (DMS) camera. The driver monitoring system (DMS) is used to detect the driver behavior by visual method, such as closing eyes, blinking, gazing direction, head movement, etc. The DMS camera can be installed at the position of the bottom of the A-pillar and facing the driver's face.

[0104] Reference Figure 4 is a structural block diagram of an optional image processing device according to the embodiment of the present application. The image processing device 40 includes:

[0105] The first acquisition unit 400 is configured to acquire the real-time position of the human eye inside the cab.

[0106] In an optional embodiment, the first acquisition unit 400 can be a third camera unit inside the vehicle (for example, the above-mentioned DMS camera).

[0107] The second acquisition unit 402 is configured to acquire the real-time external image.

[0108] In an optional embodiment, the second acquisition unit 402 can be a first camera unit outside the vehicle. The first camera unit can be a camera installed on the rearview mirror of the vehicle (including the left rearview mirror and / or the right rearview mirror) and facing the front side of the vehicle (including the left front side and / or the right front side).

[0109] The determining unit 404 is configured to determine a real-time transformation relationship between the real-time external image and the virtual image displayed on the display device inside the vehicle according to the real-time position of the human eye and the pre-acquired calibration parameter.

[0110] In an optional embodiment, the pre-acquired calibration parameter is obtained according to the calibration parameter obtaining method.

[0111] The transforming unit 406 is configured to transform the real-time external image into the virtual image according to the real-time transformation relationship.

[0112] According to the image processing device provided by the embodiment of the present application, the real-time external image of the blind area position that is blocked can be transformed into a virtual image and displayed on the display device inside the vehicle, and by obtaining the real-time position of the human eye, the virtual image can be aligned with the real-time external image while being able to move in real time following the driver's line of sight, so that the display screen picture in the driver's field of view at the current time is aligned with the external environment of the vehicle, achieving an "immersive" augmented reality experience. The image processing device only needs two camera units and one display device, does not need a three-dimensional reconstruction process, and does not need a complex camera internal and external parameter calibration, and the real-time performance depends on the lowest frame rate among the two cameras and the display device, which can meet the needs of balancing computing power, cost and user experience.

[0113] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0114] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0115] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only illustrative, and for example, the division of units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0116] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0117] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0118] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0119] The above is only the preferred embodiment of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for acquiring calibration parameters, comprising: acquiring a first image containing a first calibration pattern, the first image being an unobstructed image acquired from outside a vehicle; displaying a second image containing a second calibration pattern; acquiring M third images containing the first calibration pattern and the second calibration pattern, the third images being obstructed images acquired from inside the vehicle; determining M sets of transformation relationships between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the M third images, and the second calibration pattern contained in the M third images, M being an integer not less than 3; determining the transformation relationship between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the third image, and the second calibration pattern contained in the third image, comprising: determining a first transformation relationship between the first calibration pattern in the first image and the first calibration pattern in the third image; determining a second transformation relationship between the second calibration pattern in the third image and the second calibration pattern in the second image; determining the transformation relationship between the first image and the second image according to the first transformation relationship and the second transformation relationship.

2. The method according to claim 1, characterized by, The first image containing the first calibration pattern is acquired by using a first camera device outside the vehicle.

3. The method according to claim 2, characterized by, The first camera device is mounted on a rearview mirror of the vehicle and faces a side front of the vehicle.

4. The method according to claim 1, characterized by, The first calibration pattern is located on a calibration object at a front side of a vehicle head, and the first calibration pattern contains N feature points, N being an integer not less than 3.

5. The method according to claim 1, characterized by, After acquiring the first image containing the first calibration pattern, a de-distortion step is further included.

6. The method of claim 1, wherein The second image containing the second calibration pattern is displayed by using a display device inside the vehicle.

7. The method of claim 6, wherein The display device inside the vehicle is mounted on a vehicle A-pillar and faces a driver display.

8. The method of claim 1, wherein The third image containing the first calibration pattern and the second calibration pattern is acquired by using a second camera device inside the vehicle.

9. The method of claim 8, wherein, The second camera device is a camera simulating a human eye perspective.

10. The method of claim 1, wherein The first calibration pattern and the second calibration pattern contain not less than 3 identifiable feature points that are not collinear at different times.

11. The method of claim 10, wherein, The style of the identifiable feature points is at least one of the following: a two-dimensional code, an array of dots, and a checkerboard.

12. The method of claim 8, wherein: The acquisition of the M third images containing the first calibration pattern and the second calibration pattern includes: adjusting the second camera device inside the vehicle to M preset positions that are not collinear multiple times, and acquiring the M third images containing the first calibration pattern and the second calibration pattern.

13. The method of claim 12, wherein, The M preset positions that are not collinear are M positions in a normal range of human eye positions during driving of the driver.

14. The method of claim 8, further comprising: A third camera device is used to acquire a projection position of a principal point of the second camera device inside the vehicle on the third camera device. 15.An image processing method, comprising: acquiring a real-time position of a human eye in a driver's cabin; acquiring a real-time external image; According to the real-time position of the human eye, the pre-acquired calibration parameters, the real-time transformation relationship between the real-time external image and the virtual image displayed on the display device inside the vehicle is determined, wherein the pre-acquired calibration parameters are obtained by using any one of the calibration parameter acquisition methods in claims 1-14. According to the real-time transformation relationship, the real-time external image is transformed into the virtual image.

16. The image processing method of claim 15, wherein, The first camera device outside the vehicle is used to acquire the real-time external image.

17. The image processing method of claim 15, wherein, The third camera device inside the vehicle is used to acquire the real-time position of the human eye inside the cab.

18. A calibration parameter acquisition device, comprising: a first camera unit configured to acquire a first image containing a first calibration pattern, the first image being an all-coverage image acquired from outside the vehicle; a display unit configured to display a second image containing a second calibration pattern; a third camera unit configured to acquire M third images containing the first calibration pattern and the second calibration pattern, the third images being partial-coverage images acquired from inside the vehicle; a parameter determination unit configured to determine M sets of transformation relationships between the first image and the second image according to the first calibration pattern contained in the first image, the second calibration pattern contained in the second image, the first calibration pattern contained in the M third images, and the second calibration pattern contained in the M third images, M being an integer not less than 3; the parameter determination unit comprises: a first determination sub-unit configured to determine a first transformation relationship between the first calibration pattern in the first image and the first calibration pattern in the third images; a second determination sub-unit configured to determine a second transformation relationship between the second calibration pattern in the third images and the second calibration pattern in the second image; a third determination sub-unit configured to determine the transformation relationship between the first image and the second image according to the first transformation relationship and the second transformation relationship.

19. The device according to claim 18, characterized in that The first camera unit is mounted outside the vehicle.

20. The device of claim 19, wherein, The first camera unit is mounted on the side mirror of the vehicle and faces the side front of the vehicle.

21. The device of claim 18, wherein, The first calibration pattern is located on a calibration object on the side front of the vehicle head, and the first calibration pattern contains N feature points, N being an integer not less than 3.

22. The parameter calibration acquisition apparatus according to claim 21, characterized by, The N feature points of the first calibration pattern are completely covered by the field of view of the first camera unit.

23. The calibration parameter acquisition device according to claim 18, further comprising a processing unit configured to de-distort the acquired first image containing the first calibration pattern.

24. The device according to claim 18, wherein The display device is mounted inside the vehicle.

25. The device of claim 24, wherein, The display device is mounted on the A-pillar of the vehicle and faces the driver for display.

26. The device of claim 18, wherein The second camera unit is mounted inside the vehicle.

27. The device of claim 26, wherein, The second camera unit is a camera simulating the perspective of the human eye.

28. The device of claim 18, wherein, The first calibration pattern and the second calibration pattern contain not less than 3 identifiable feature points that are not collinear at the same time.

29. The device of claim 28, wherein, The style of the identifiable feature points is at least one of the following: a two-dimensional code, an array of dots, and a checkerboard.

30. The device of claim 18, wherein, The third camera unit is used to acquire M third images containing the first calibration pattern and the second calibration pattern, which comprises adjusting the second camera unit inside the vehicle to M preset positions not in a same line multiple times, and acquiring M third images containing the first calibration pattern and the second calibration pattern.

31. The device of claim 30, wherein, The M preset positions not in a same line are M positions in a normal eye position change range of a driver during driving.

32. The parameter estimation apparatus according to claim 18, further comprising: The third camera unit is used to acquire a projection position of an optical center of the second camera unit inside the vehicle on the third camera unit.

33. A storage medium, characterized by The storage medium comprises a stored program, wherein the program controls a device where the storage medium is located to execute the calibration parameter acquisition method in any one of claims 1 to 14 when the program is running.

34. An electronic device, comprising: comprise: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the calibration parameter acquisition method in any one of claims 1 to 14 via executing the executable instructions.

Images