Panoramic image calibration device, system, method, computer system and storage medium

Abstract

The present application provides a panoramic image calibration device, system, method, computer system and storage medium. The device includes at least a support frame, an adjustment mechanism and a plurality of checkerboards. The support frame is arranged parallel to the working ground and is located above the working ground. One end of the adjustment mechanism is connected to the support frame, and the other end is connected to the plurality of checkerboards. The adjustment mechanism can drive the plurality of checkerboards to move. The present application scheme sets the checkerboards to be movable. When not in use, the checkerboards are raised. When calibration is required, the checkerboards are adjusted to the required height below the front, rear, left and right cameras of the vehicle to be calibrated by the adjustment mechanism before calibration. The height can be switched in real time according to different types of vehicles to ensure that the position of the checkerboard is at an appropriate height, thereby obtaining more accurate image data, so that the position and direction of the checkerboards can be identified by subsequent processing of the vehicle image to achieve the purpose of calibration.

Description

Technical Field

[0001] The present application relates to the field of automobile driving assistance technology, and in particular to a panoramic image calibration device, system, method, computer system and storage medium. Background Art

[0002] With the continuous development and advancement of the automotive industry, vehicle image processing has become a popular research area. Among them, vehicle panoramic image calibration is an important part of vehicle image processing, which is mainly used to improve automobile safety performance, intelligent driving and other applications.

[0003] With the continuous development and advancement of the automotive industry, vehicle image processing has become a popular research area. Among them, vehicle panoramic image calibration is an important part of vehicle image processing, which is mainly used to improve automobile safety performance, intelligent driving and other applications.

[0004] At present, there are two main solutions for vehicle panoramic image calibration. The first is to use a sprayed checkerboard for calibration. During the calibration process, the checkerboard needs to be sprayed on the ground in different parts of the vehicle, and then the checkerboard image is collected by the panoramic camera for calibration. The second is to use four projectors installed on the front, rear, left and right sides of the vehicle above the ceiling. The projector solution requires a large space for projection, and the ground must not be dirty. Therefore, during the calibration process, this solution requires a large space around the vehicle and requires the support of a projector, which is costly.

[0005] In addition, there are some other problems with the existing panoramic image calibration scheme. For example, if a spray-painted chessboard is used for calibration, the spray-painted chessboard is easily worn due to the continuous rolling of the vehicle, resulting in incomplete images and affecting the calibration effect. At the same time, the maintenance cost is also very high, and the site may reflect light, causing calibration failure. Secondly, if a four-projector solution is used, a large area of ​​the site needs to be vacated, which is also affected by reflections like the spray-painted chessboard, and the projector is expensive, which increases the cost of the entire calibration system. At the same time, different models may require different sites and different projectors, which increases the complexity of the calibration system.

[0006] Therefore, a new panoramic image calibration solution is needed to solve the above problems. Summary of the invention

[0007] In order to solve one or more of the above-mentioned technical problems existing in the prior art, the embodiments of the present application provide a new panoramic image calibration device, system, method, computer system and storage medium to solve the problems existing in the prior art.

[0008] In order to achieve the above-mentioned purpose, the technical solution adopted by this application to solve its technical problems is:

[0009] In a first aspect, the present application provides a panoramic image calibration device, the device comprising at least a support frame, an adjustment mechanism, and a plurality of checkerboards;

[0010] The support frame is arranged parallel to the working ground and is located above the working ground;

[0011] One end of the adjustment mechanism is connected to the support frame, and the other end is connected to the multiple chessboard grids. The adjustment mechanism can drive the multiple chessboard grids to move.

[0012] In a specific embodiment, the chessboard includes four grids, and the four chessboard grids are respectively located around the vehicle to be calibrated when in use.

[0013] In a specific embodiment, the adjustment mechanism includes a lifting module, which is arranged vertically relative to the support frame. The multiple checkerboard grids are connected to one end of the lifting module away from the support frame, and the lifting module can drive the multiple checkerboard grids to reciprocate in a direction perpendicular to the support frame.

[0014] In a specific embodiment, the adjustment mechanism also includes a guide rail, which is arranged on the support frame along its horizontal direction, and the lifting module is connected to the guide rail at one end away from the multiple checkerboard grids, and the lifting module can move back and forth on the guide rail in a direction parallel to the support frame.

[0015] In a specific embodiment, the device further comprises:

[0016] The control mechanism is connected to the adjusting mechanism and is used to control the adjusting mechanism to drive the multiple chessboard grids to move.

[0017] In a second aspect, corresponding to the panoramic image calibration device described above, the present application further provides a panoramic image calibration system, the system comprising the panoramic image calibration device described above, and the system further comprising:

[0018] An image acquisition module, used for acquiring a plurality of images to be processed including the plurality of checkerboards and the vehicle to be calibrated;

[0019] A parameter calculation module, used to perform calibration calculation according to the image to be processed and the heights of the plurality of chessboard grids relative to the working ground, to obtain calibration internal and external parameters;

[0020] The image stitching module is used to stitch the multiple images to be processed according to the calibration internal and external parameters to obtain a panoramic surround image of the vehicle to be calibrated.

[0021] In a specific embodiment, the image acquisition module includes a plurality of fisheye cameras, the number of the fisheye cameras corresponds to the number of the chessboard grids, and the installation position of each of the fisheye cameras corresponds to one of the chessboard grids.

[0022] In a third aspect, corresponding to the above-mentioned panoramic image calibration system, the present application also provides a panoramic image calibration method, the method comprising:

[0023] Using an adjustment mechanism to adjust the positions of a plurality of checkerboard grids relative to the vehicle to be calibrated according to the type of the vehicle to be calibrated parked on the working ground;

[0024] Acquire a plurality of images to be processed including the plurality of checkerboards and the vehicle to be calibrated;

[0025] Performing calibration calculation according to the image to be processed and the heights of the plurality of checkerboard grids relative to the working ground to obtain calibration internal and external parameters;

[0026] The plurality of images to be processed are stitched together according to the calibration internal and external parameters to obtain a panoramic surround image of the vehicle to be calibrated.

[0027] In a fourth aspect, a computer system is also provided, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the panoramic image calibration method is implemented.

[0028] In a fifth aspect, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the panoramic image calibration method is implemented.

[0029] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0030] The panoramic image calibration device, system, method, computer system and storage medium provided in the embodiments of the present application, the device at least includes a support frame, an adjustment mechanism and a plurality of checkerboards, the support frame is arranged parallel to the working ground and is located above the working ground, one end of the adjustment mechanism is connected to the support frame, and the other end is connected to the plurality of checkerboards, and the adjustment mechanism can drive the plurality of checkerboards to move. The present application scheme arranges the checkerboards to be movable, and the checkerboards are raised when not in use, and when calibration is required, the checkerboards are adjusted to the required height below the front, rear, left and right cameras of the vehicle to be calibrated through the adjustment mechanism before calibration. This can achieve real-time switching of heights according to different types of vehicles to ensure that the position of the checkerboard is at an appropriate height, thereby obtaining more accurate image data, so that the position and direction of the checkerboards can be identified by subsequent processing of vehicle images to achieve the purpose of calibration;

[0031] Furthermore, the panoramic image calibration device, system, method, computer system, and storage medium provided in the embodiments of the present application, compared with the existing checkerboard spraying and projector solutions, use a lifting checkerboard, do not need to clear a large area around the vehicle, and do not need to use a large projector, so the calibration process is simpler and faster;

[0032] Furthermore, the panoramic image calibration device, system, method, computer system and storage medium provided in the embodiments of the present application can be raised at any time and will not be worn out due to vehicle rolling, thereby reducing the cost of maintenance;

[0033] Furthermore, the panoramic image calibration device, system, method, computer system and storage medium provided in the embodiments of the present application, because the lifting chessboard can ensure that the distance between the chessboard and the camera is always the same during calibration, avoids calibration errors caused by problems such as projection angle, chessboard wear, reflection and dirt in existing solutions.

[0034] Not all products in this application need to have all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a schematic diagram of the three-dimensional structure of a panoramic image calibration device provided in an embodiment of the present application;

[0037] Figure 2is a front view of a panoramic image calibration device provided in an embodiment of the present application;

[0038] Figure 3 It is a left view of the panoramic image calibration device provided in an embodiment of the present application;

[0039] Figure 4 is a top view of a panoramic image calibration device provided in an embodiment of the present application;

[0040] Figure 5 is a structural schematic diagram of a panoramic image calibration system provided in an embodiment of the present application;

[0041] Figure 6 is a flow chart of a panoramic image calibration method provided in an embodiment of the present application;

[0042] Figure 7 It is a schematic diagram of the catadioptric fisheye camera model and coordinate system;

[0043] Figure 8 It is a schematic diagram of the relationship between the chessboard position and the working ground;

[0044] Fig. 9 It is a schematic diagram of the unified coordinate system established in image stitching;

[0045] Fig.10 It is an architectural diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] As described in the background technology, in the existing vehicle panoramic image calibration scheme, if a sprayed chessboard is used for calibration, the sprayed chessboard is easily worn due to the continuous rolling of the vehicle, resulting in an incomplete image, affecting the calibration effect, and the maintenance cost is also very high, and the site may reflect light, resulting in calibration failure. If a four-projector solution is used, a large area of ​​the site needs to be vacated, which is also affected by reflections like the sprayed chessboard, and the projector is expensive, increasing the cost of the entire calibration system. At the same time, different models may require different sites and different projectors, increasing the complexity of the calibration system.

[0048] In order to solve one or more technical problems in the prior art, the present application creatively proposes a new panoramic image calibration device. By setting the chessboard movable, the chessboard is raised when not in use. When calibration is needed, the chessboard is adjusted to the required height below the front, rear, left, and right cameras of the vehicle to be calibrated through an adjustment mechanism and then calibration is performed. Real-time height switching can be achieved according to different types of vehicles to ensure that the chessboard is at an appropriate height, thereby obtaining more accurate image data, so that the position and direction of the chessboard can be identified by subsequent processing of the vehicle image to achieve the purpose of calibration.

[0049] The scheme of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0050] Embodiment 1

[0051] Reference Figures 1 to 4 As shown, the panoramic image calibration device provided in the embodiment of the present application generally includes a support frame 100, an adjustment mechanism 200 and a plurality of checkerboards 300. Among them, the support frame 100 is arranged parallel to the working ground and is located above the working ground, and the working ground is used to park the vehicle to be calibrated. One end of the adjustment mechanism 200 is connected to the support frame 100, and the other end is connected to the plurality of checkerboards 300. The adjustment mechanism 200 is configured to drive the movement of the plurality of checkerboards 300 to ensure that the position of the checkerboards 300 matches the vehicle to be calibrated, so as to obtain more accurate image data, so as to identify the position and direction of the checkerboards by processing the vehicle image later, and achieve the purpose of calibration. It can be understood that since the calibration is performed in the form of a checkerboard in the present application scheme, there is no need to clear a large area for projection, which can save working space and reduce maintenance costs.

[0052] It is understandable that the adjustment mechanism 200 is connected to the support frame 100 and is also located above the working ground. The chessboard 300 is connected to the adjustment mechanism 200. When not in use, the chessboard 300 is raised by the adjustment mechanism 200 and stored in the space above the working ground. When calibration is required, the chessboard is lowered to the required height below the cameras around the vehicle to be calibrated by the adjustment mechanism 200 for calibration.

[0053] Further references Figure 1As shown, as a preferred embodiment, in the embodiment of the present application, the support frame 100 includes a first support rod 110, a second support rod 120 and a third support rod 130. The first support rod 110 is arranged vertically relative to the working ground, the second support rod 120 and the third support rod 130 are arranged parallel to the working ground, and the second support rod 120 and the third support rod 130 are arranged vertically relative to each other. As a preferred embodiment, in the embodiment of the present application, the second support rod 120 and the third support rod 130 are connected to form a rectangular frame. In specific implementation, the number of the first support rod 110 includes 4, the number of the second support rod 120 includes 2, and the number of the third support rod 130 includes 3. The four first support rods 110 are respectively connected at the four corners of the rectangular frame, and are used to fix the rectangular frame above the working ground, such as fixing the rectangular frame on the roof above the working ground. The two second support rods 120 are arranged parallel to each other, the three third support rods 130 are all disposed between the two second support rods 120 and arranged parallel to each other, and both ends of the three third support rods 130 are respectively connected to one of the second support rods 120 .

[0054] Further references Figures 1 to 3 As shown, as a preferred embodiment, in the embodiment of the present application, the adjustment mechanism 200 includes a lifting module 210. Exemplarily, the lifting module 210 can adopt a lifting platform. The lifting module 210 is arranged vertically relative to the support frame 100, one end of the lifting module 210 is connected to the support frame 100, and the end of the lifting module 210 away from the support frame 100 is connected to the chessboard 300. The lifting module 210 can drive the chessboard 300 to reciprocate along the direction 100 perpendicular to the support frame. Preferably, the number of lifting modules 210 is the same as the number of chessboards 300, and each lifting module 210 is connected to a chessboard 300 at one end away from the support frame 100. This arrangement can make the position of each chessboard 300 along the direction 100 perpendicular to the support frame individually adjustable, thereby further improving the applicability of the panoramic image calibration device, so that it can be applied to various types of vehicles.

[0055] In order to further improve the applicability of the panoramic image calibration device, so that the adjustment mechanism 200 can move the chessboard 300 to different positions according to different types of vehicles, the adjustment mechanism 200 in the embodiment of the present application further includes a guide rail (not shown). The guide rail is arranged on the support frame 100 along its horizontal direction, and the end of the lifting module 210 away from the chessboard 300 is connected to the guide rail, and the lifting module 210 is configured to be able to reciprocate on the guide rail in a direction parallel to the support frame.

[0056] As a preferred implementation, in the embodiment of the present application, the number of guide rails includes 5. In a specific embodiment, two of the guide rails are respectively arranged on one of the second support rods 120 and are adapted to the second support rod 120, and the other three guide rails are respectively arranged on one of the third support rods 130 and are adapted to the second support rod 120. This arrangement allows the corresponding lifting module 210 to reciprocate along the corresponding guide rail on the second support rod 120 or the third support rod 130, thereby driving the corresponding chessboard 300 to reciprocate along the corresponding guide rail on the second support rod 120 or the third support rod 130, thereby realizing the change of the position of the chessboard 300 in the direction of the support frame 100.

[0057] It is understandable that in the embodiment of the present application, the guide rail can be a separate component, and each guide rail is respectively connected to one of the second support rods 120 or one of the third support rods 130. The guide rail can also be integrally formed with the second support rod 120 or the third support rod 130, that is, the guide rail and the second support rod 120 or the third support rod 130 are an integral structure, which can realize the function of the guide rail so that the lifting module 210 can slide along it, and can also realize the function of the second support rod 120 or the third support rod 130 to provide support for the lifting module 210.

[0058] As a preferred implementation, in the embodiment of the present application, the number of the chessboard 300 includes four, and the four chessboards 300 are respectively located around the vehicle to be calibrated when in use, that is, the four chessboards 300 are respectively located at the front, rear, left, and right of the vehicle to be calibrated when in use. Preferably, the four chessboards 300 are respectively located under the front, rear, left, and right cameras of the vehicle to be calibrated when in use, so as to facilitate the camera to collect images. It can be understood that when the number of the chessboard 300 is four, the number of the lifting modules 210 is also four.

[0059] It should be noted that the number of four checkerboard grids 300 is merely an exemplary and non-restrictive description. Without violating the inventive concept of the present application, the number of checkerboard grids 300 can be set according to actual needs and is not specifically limited here.

[0060] It is understandable that during the calibration process, in order to improve the accuracy of the calibration, on the one hand, the chessboard needs to meet the requirements of the flatness of the chessboard surface and the size of the chessboard surface to ensure that the panoramic effective display area is covered as much as possible; on the other hand, after the adjustment mechanism drives the chessboard to move to the required position, it is necessary to ensure that the chessboard is placed parallel to the working ground.

[0061] As a preferred implementation, in the embodiment of the present application, the device also includes a control mechanism (not shown), which is connected to the adjustment mechanism 200 and is used to control the adjustment mechanism 200 to drive the multiple chessboards 300 to move. In a specific embodiment, the control mechanism can calculate the position information of each chessboard 300 according to the type of vehicle to be calibrated, and then control the adjustment mechanism 200 to move each chessboard 300 to a corresponding position according to the position information, so as to improve the automation of the panoramic image calibration device. The specific implementation method of the control mechanism is not specifically limited here. Without violating the inventive concept of the present application, any known control mechanism can be used in the embodiment of the present application.

[0062] Embodiment 2

[0063] Corresponding to the above embodiment 1, the present application also provides a panoramic image calibration system, referring to Figure 5 As shown, the system includes the panoramic image calibration device 10, the image acquisition module 20, the parameter calculation module 30 and the image stitching module 40 described in the first embodiment. The relevant contents of the panoramic image calibration device 10 can be referred to the above introduction, and will not be described in detail later.

[0064] As a preferred implementation, the panoramic image calibration system in the embodiment of the present application uses image processing technology to identify the position and direction of the chessboard by processing the vehicle image, thereby achieving the purpose of calibration.

[0065] In a specific implementation, the image acquisition module is configured to acquire multiple images to be processed including multiple checkerboards and a vehicle to be calibrated. The parameter calculation module is configured to perform calibration calculations based on the images to be processed acquired by the image acquisition module and the heights of the multiple checkerboards relative to the working ground to obtain calibration internal and external parameters. The image stitching module is configured to stitch multiple images to be processed based on the calibration internal and external parameters calculated by the parameter calculation module to obtain a panoramic surround image of the vehicle to be calibrated.

[0066] As a preferred implementation, in the embodiment of the present application, the image acquisition module includes a plurality of fisheye cameras, the number of which corresponds to the number of checkerboards, and the installation position of each fisheye camera corresponds to one of the checkerboards. Preferably, the number of checkerboards is four, and the number of fisheye cameras is also four. When in use, the four fisheye cameras are respectively located above the front, rear, left, and right of the vehicle to be calibrated, and the four checkerboards 300 are respectively located below the four fisheye cameras at the front, rear, left, and right of the vehicle to be calibrated.

[0067] Embodiment 3

[0068] Corresponding to the above-mentioned embodiment 2, the present application further provides a panoramic image calibration method, which is based on the panoramic image calibration system described in embodiment 1, wherein, in this embodiment, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be described in detail later. Figure 6 As shown, the method comprises the following steps:

[0069] S100: Using an adjustment mechanism to adjust positions of a plurality of checkerboard grids relative to the vehicle to be calibrated according to the type of the vehicle to be calibrated parked on the working ground.

[0070] Specifically, the vehicle to be calibrated is parked on the working ground, the type of the vehicle to be calibrated is obtained, and the position of each chessboard grid relative to the vehicle to be calibrated is adjusted by using an adjustment mechanism according to the type of the vehicle to be calibrated, and each chessboard grid is moved to an appropriate position.

[0071] S200: Acquire a plurality of images to be processed including the plurality of checkerboards and the vehicle to be calibrated.

[0072] Specifically, an image acquisition module (such as a fisheye camera) is used to acquire a plurality of images to be processed, wherein each image to be processed should include a corresponding chessboard and a vehicle to be calibrated.

[0073] S300: performing calibration calculation according to the image to be processed and the heights of the plurality of checkerboard grids relative to the working ground to obtain calibration internal and external parameters.

[0074] Specifically, since the position of the chessboard in the present application scheme will be adjusted to accommodate different types of vehicles to be calibrated, the height of the chessboard relative to the working ground needs to be included in the calculation factors when calculating the internal and external calibration parameters to improve the accuracy of the calibration.

[0075] S400: stitching the multiple images to be processed according to the calibration internal and external parameters to obtain a panoramic surround image of the vehicle to be calibrated.

[0076] Specifically, in the embodiment of the present application, the coordinates of a point in the panoramic image area in the unified coordinate system and the size of the panoramic view effective area are first used to determine which camera the original image of this point comes from, and then the coordinate value on the overhead image is calculated by the reverse projection of the above relationship for this camera, and finally the corresponding original image coordinates are obtained by the reverse projection transformation from the overhead image to the original image. After traversing the entire panoramic area and interpolating the original fisheye image coordinate data obtained by reverse projection, the panoramic view image can be obtained.

[0077] The following uses a fisheye camera as an example to illustrate the calibration implementation.

[0078] Step 1: Fisheye camera model and calibration method.

[0079] Specifically, after collecting the image to be processed, the coordinates of the corner points of the image to be processed are extracted to participate in the calibration calculation, such as Figure 7 As shown in the figure, the projection relationship from the world coordinates to the imaging surface coordinates in this model is a nonlinear projection function containing the camera distortion parameters, which is expressed by a Taylor series expansion polynomial as follows:

[0080] f(ρ)=a0+a1ρ+a2ρ 2 +…+a n ρ n .

[0081] in, Represents the distance from the projection point to the center of the imaging surface, a0, a1, ..., a n Represents the coefficients of the polynomial, which characterizes the camera distortion parameters.

[0082] Step 2: Calculate and calibrate internal and external parameters.

[0083] Specifically, assuming that Rt = [r1 r2 r3 t] represents the rotation and translation matrix from world coordinates to camera coordinates, that is, the external parameter matrix of the camera's installation position relative to the chessboard, and P (x, y, z) is an object point in the world coordinate system, then the projection relationship from P (x, y, z) to its imaging point is expressed as:

[0084]

[0085] Without loss of generality, the coordinates of the imaging point and the pixel coordinates on the image to be processed have the following affine transformation relationship:

[0086]

[0087] Among them, c, e, and d all represent the coefficients of the radiation transformation relationship.

[0088] Therefore, the correspondence between the image coordinates and the world coordinates can be determined through a series of parameters. When obtaining these parameters, the camera's external parameters and the initial values ​​of the corresponding internal parameters are calculated using the above projection relationship and the corner point coordinate data of the template identified from the image to be processed, under the assumption that the center of the photosensitive surface is consistent with the center of the image, and then the Levenbereg-Marquardt algorithm is used to iteratively solve the maximum likelihood estimate of the imaging projection error value for the corner point coordinates in the template, and the pseudo-inverse solution of the overdetermined equations is used to obtain accurate camera internal and external parameters. The above calculation process is not the invention of this application, and the relevant calculation process of the prior art can be referred to. It will not be repeated here.

[0089] Step 3: Conversion of the relationship between the chessboard and the working ground.

[0090] Specifically, refer to Figure 8 As shown, the size distance H between the chessboard and the ground is known. Based on the above model and the law of camera pinhole imaging, the corresponding relationship between the chessboard and the ground is obtained as follows:

[0091]

[0092] Where h represents the distance of the chessboard from the center of the camera coordinate system in the vertical direction, H P is the distance between the raised parallel chessboard and the ground; X w1 , Y w1 , Z w1 is the direction vector of the chessboard's coordinate system, X w , Y w , Z w is the direction vector of the camera's projection coordinate system on the ground, t i is the coefficient.

[0093] According to the calibration extrinsic parameters calculated in step 2, the camera extrinsic parameter matrix is ​​obtained as follows:

[0094] R t =[r1 r2 r3 t].

[0095] Among them, r1 and r2 are the direction vectors of the X and Y axes in the chessboard coordinate system in the camera coordinate system. The plane equation determined by the X and Y axes can be obtained through the direction vectors, and the distance between the origin of the camera coordinate system and the XY plane is calculated as the camera height h.

[0096] Step 4: Image stitching based on calibrated internal and external parameters.

[0097] First, according to the camera installation position in the system design, establish Fig. 9 The unified coordinate system shown in Figure 2. Among them, W, H, L1, L2, L3, and L4 are the dimensions of the effective area of ​​the panoramic view (unit: mm). The coordinates of the projection points of the front, left, right, and rear cameras on the ground in the unified coordinate system (OXY) are O F (Cfx,Cfy),O L (Clx, Cly), O R (Crx,Cry),O B (Cbx, Cby). They can be expressed as the following translation vectors:

[0098]

[0099]

[0100]

[0101]

[0102] Secondly, the projection relationship is analyzed. For camera i (i = front, back, left, right), the conversion process from the original image of the side camera to each coordinate system in the panoramic image projection in the image stitching is as follows:

[0103] First, rotate the camera coordinate system (Oc-Xc-Yc-Zc) of the i-side camera so that the Z axis is vertically downward, and the X-axis and Y-axis directions are aligned with Fig. 9 In the (oi-xy) coordinate system, the x and y directions are consistent. Ri is used to represent the rotation matrix, that is, the external parameter matrix. Its transformation relationship can be expressed as

[0104]

[0105] The rotated camera coordinate system (oi-xy) and the unified coordinate system (OXY) have the following corresponding relationship:

[0106]

[0107] Where ti represents the translation vector, R0 represents the rotation matrix between the two coordinate systems, and its expression is:

[0108]

[0109] As a preferred example, for the front, left, rear, and right cameras, the values ​​of the rotation angle g are: -90°, 0°, 180°, and 90°, respectively.

[0110] It should be noted here that the embodiment of the present application does not limit the specific calculation process of each step in the calibration process. Under the premise of not violating the inventive concept of the present application, the user can set it according to actual needs.

[0111] Embodiment 4

[0112] Corresponding to the above-mentioned embodiment three, the present application also provides a computer system, including: a processor and a memory, the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the panoramic image calibration method provided by any one of the above-mentioned embodiments is executed.

[0113] in, Fig.10The computer system 1500 is shown as an example, and may include a processor 1510, a video display adapter 1511, a disk drive 1512, an input / output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, the video display adapter 1511, the disk drive 1512, the input / output interface 1513, the network interface 1514, and the memory 1520 may be communicatively connected via a communication bus 1530.

[0114] Among them, the processor 1510 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided by the present invention.

[0115] The memory 1520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1520 can store an operating system 1521 for controlling the operation of the electronic device and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device. In addition, a web browser 1523, a data storage management system 1524, and a device identification information processing system 1525, etc. can also be stored. The above-mentioned device identification information processing system 1525 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present invention. In short, when the technical solution provided by the present invention is implemented by software or firmware, the relevant program code is stored in the memory 1520 and is called and executed by the processor 1510.

[0116] The input / output interface 1513 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0117] The network interface 1514 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0118] The bus includes a path that transmits information between the various components of the device (eg, processor 1510, video display adapter 1511, disk drive 1512, input / output interface 1513, network interface 1514, and memory 1520).

[0119] In addition, the electronic device can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment, etc.

[0120] It should be noted that, although the above device only shows a processor 1510, a video display adapter 1511, a disk drive 1512, an input / output interface 1513, a network interface 1514, a memory 1520, a bus, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present invention, and does not necessarily include all the components shown in the figure.

[0121] Embodiment 5

[0122] Corresponding to the above-mentioned embodiment three, the embodiment of the present application also provides a computer-readable storage medium, wherein, in this embodiment, the same or similar contents as those in the above-mentioned embodiment three can be referred to the above introduction and will not be repeated later.

[0123] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the panoramic image calibration method as described above is implemented.

[0124] In some implementations, in the embodiments of the present application, when the computer program is executed by the processor, the steps corresponding to the method described in Embodiment 1 may also be implemented. Please refer to the detailed description in Embodiment 1 and no further details will be given here.

[0125] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0126] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0127] In the description of the present application, it should be understood that the terms "vertical", "parallel", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "multiple" means two or more.

[0128] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0129] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A panoramic image calibration device, characterized in that: The device at least comprises a support frame, an adjustment mechanism and a plurality of checkerboard grids; The support frame is arranged parallel to the working ground and is located above the working ground; One end of the adjustment mechanism is connected to the support frame, and the other end is connected to the plurality of checkerboards, and the adjustment mechanism can drive the plurality of checkerboards to move; The chessboard includes four grids, and the four chessboard grids are respectively located around the vehicle to be calibrated when in use; Wherein, after the adjustment mechanism drives the plurality of checkerboard grids to move to the required positions, the plurality of checkerboard grids are placed parallel to the working ground; The support frame includes a first support rod, a second support rod and a third support rod, the first support rod is arranged vertically relative to the working ground, the second support rod and the third support rod are arranged parallel to the working ground, and the second support rod and the third support rod are arranged vertically relative to each other; The adjustment mechanism includes a lifting module and a guide rail, the lifting module is arranged vertically relative to the support frame, the multiple chessboards are connected to one end of the lifting module away from the support frame, and the lifting module can drive the multiple chessboards to reciprocate in a direction perpendicular to the support frame; The guide rail is arranged on the support frame along its horizontal direction, part of the guide rail is arranged on the second support rod, and part of the guide rail is arranged on the third support rod. The lifting module is connected to the guide rail at one end away from the multiple checkerboard grids, and the lifting module can move back and forth on the guide rail in a direction parallel to the support frame.

2. The panoramic image calibration device according to claim 1, characterized in that: The device also includes: The control mechanism is connected to the adjusting mechanism and is used to control the adjusting mechanism to drive the multiple chessboard grids to move.

3. A panoramic image calibration system, characterized in that: The system comprises the panoramic image calibration device according to any one of claims 1 or 2, and the system further comprises: An image acquisition module, used for acquiring a plurality of images to be processed including the plurality of checkerboards and the vehicle to be calibrated; A parameter calculation module, used to perform calibration calculation according to the image to be processed and the heights of the plurality of chessboard grids relative to the working ground, to obtain calibration internal and external parameters; The image stitching module is used to stitch the multiple images to be processed according to the calibration internal and external parameters to obtain a panoramic surround image of the vehicle to be calibrated.

4. The panoramic image calibration system according to claim 3, characterized in that: The image acquisition module includes a plurality of fisheye cameras, the number of the fisheye cameras corresponds to the number of the chessboard grids, and the installation position of each of the fisheye cameras corresponds to one of the chessboard grids.

5. A panoramic image calibration method, characterized in that: The method is based on the panoramic image calibration system according to claim 3 or 4, and the method comprises: Using an adjustment mechanism to adjust the positions of a plurality of checkerboard grids relative to the vehicle to be calibrated according to the type of the vehicle to be calibrated parked on the working ground; Acquire a plurality of images to be processed including the plurality of checkerboards and the vehicle to be calibrated; Performing calibration calculation according to the image to be processed and the heights of the plurality of checkerboard grids relative to the working ground to obtain calibration internal and external parameters; The plurality of images to be processed are stitched together according to the calibration internal and external parameters to obtain a panoramic surround image of the vehicle to be calibrated.

6. A computer system, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the panoramic image calibration method described in claim 5 is implemented.

7. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed, the panoramic image calibration method described in claim 5 is implemented.

Images