Camera-based bird's eye view calibration method, device, and storage medium

Abstract

The application discloses a camera-based bird's-eye view calibration method, device and storage medium. The method is applied to a movable device including a camera and a laser device. The method comprises the following steps: acquiring a calibration image collected by the camera on a calibration chessboard and a baffle laser point cloud collected by the laser device on a preset baffle; the calibration chessboard comprises an identification code, and the preset baffle is arranged at the edge of the chessboard and is perpendicular to the plane of the chessboard; performing bird's-eye view calibration processing according to the pixel coordinates and real coordinates of the identification code to obtain an initial bird's-eye view transformation matrix; acquiring an initial bird's-eye view of the calibration chessboard according to the initial bird's-eye view transformation matrix, and acquiring an edge image point cloud of the edge in the initial bird's-eye view; determining a bird's-eye view vehicle body transformation matrix according to the edge image point cloud, the baffle laser point cloud and a laser external parameter; and determining a target bird's-eye view transformation matrix according to the acquired bird's-eye view visual range, a preset scaling ratio and the bird's-eye view vehicle body transformation matrix. The above scheme can improve the bird's-eye view calibration efficiency of the camera.

Description

Technical Field

[0001] This application relates to the field of camera calibration technology, and in particular to a camera-based bird's-eye view calibration method, device, and storage medium. Background Technology

[0002] Camera calibration is a technique that calculates and determines the parameters of the camera's imaging geometric model. It is used to establish the correspondence between three-dimensional space and two-dimensional image coordinates and is a key step in three-dimensional modeling, image measurement, and machine vision.

[0003] For example, in the driving scenarios of mobile devices, taking mobile robots as an example, a bird's-eye view (BEV) is a commonly used method for representing their environment. A BEV scene can be constructed based on camera data from one or more perspectives, which involves the accurate calibration of the bird's-eye view.

[0004] However, existing calibration methods typically require designing a large calibration cloth that surrounds the device on all four sides, based on the actual size of the mobile device. Furthermore, the mobile device needs to be precisely positioned at the calibration location during the calibration process, which is very inconvenient and makes it difficult to handle rapid recalibration of the mobile device after changing cameras or adjusting its mechanical structure. Summary of the Invention

[0005] This application provides at least one camera-based bird's-eye view mapping method, apparatus, device, and computer-readable storage medium.

[0006] This application provides a camera-based bird's-eye view calibration method, applied to a mobile device including a camera and a pre-calibrated laser device, comprising: acquiring a calibration image obtained by the camera capturing images of a preset calibration checkerboard and a laser point cloud of a preset baffle acquired by the laser device; the calibration checkerboard includes an identification code, and the preset baffle is disposed on the edge of the calibration checkerboard and perpendicular to the plane of the calibration checkerboard; performing bird's-eye view calibration processing based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard to obtain an initial bird's-eye view transformation matrix; acquiring an initial bird's-eye view of the calibration checkerboard based on the initial bird's-eye view transformation matrix, and acquiring edge image point clouds of the edges in the initial bird's-eye view; determining a bird's-eye view vehicle transformation matrix based on the edge image point clouds, the baffle laser point cloud, and the pre-calibrated laser extrinsic parameters of the laser device; and determining a target bird's-eye view transformation matrix based on the visible range of the acquired bird's-eye view, a preset scaling ratio, and the bird's-eye view vehicle transformation matrix.

[0007] In one embodiment, before performing bird's-eye view calibration processing based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard to obtain an initial bird's-eye view transformation matrix, the method further includes: identifying the identification code corner points of the identification code in the calibration image and obtaining the pixel coordinates of the identification code corner points; determining the actual coordinates of the identification code corner points in the calibration checkerboard based on the pixel coordinates, the actual scale between the calibration image and the calibration checkerboard, and the checkerboard size information of the calibration checkerboard.

[0008] In one embodiment, the step of performing bird's-eye view calibration processing based on the pixel coordinates of the identifier in the calibration image and the actual coordinates of the identifier in the calibration checkerboard to obtain an initial bird's-eye view transformation matrix includes: obtaining a preset scaling ratio and the visible range of the bird's-eye view; converting the actual coordinates into bird's-eye view image coordinates based on the scaling ratio and the visible range of the bird's-eye view; and determining the initial bird's-eye view transformation matrix based on the pixel coordinates of the identifier corner points and the bird's-eye view image coordinates.

[0009] In one embodiment, determining the initial bird's-eye view transformation matrix based on the pixel coordinates of the identification code corner points and the bird's-eye view image coordinates includes: acquiring the identification code corner points of multiple identification codes; selecting multiple target identification code corner points from the multiple identification code corner points according to a preset random sampling consensus algorithm; determining multiple initial transformation matrices and the reprojection error of each initial transformation matrix based on the pixel coordinates of the multiple target identification code corner points and the bird's-eye view image coordinates; and determining the initial bird's-eye view transformation matrix from the multiple initial transformation matrices based on the reprojection error.

[0010] In one embodiment, obtaining an initial bird's-eye view of the calibration checkerboard based on the initial bird's-eye view transformation matrix, and obtaining an edge image point cloud of the edges in the initial bird's-eye view, includes: transforming the calibration image according to the initial bird's-eye view transformation matrix to obtain the initial bird's-eye view; identifying designated corner points in the initial bird's-eye view; determining the edge corner points of the calibration checkerboard in the initial bird's-eye view based on the checkerboard size information and the designated corner points; and selecting line segment points based on the baffle size information of the preset baffle, the edge corner points, and a preset distance to obtain the edge image point cloud.

[0011] In one embodiment, the step of selecting line segment points based on the baffle size information of the preset baffle, the edge corner points, and the preset distance to obtain the edge image point cloud includes: transforming the coordinates of the edge corner points according to the initial bird's-eye view transformation matrix to obtain the real edge corner points; determining the baffle edge points according to the preset linear interpolation algorithm, the real edge corner points, and the baffle size information; and selecting the line segment points from the line segments between the baffle edge points and the real edge corner points according to the preset distance to obtain the edge image point cloud.

[0012] In one embodiment, determining the bird's-eye view vehicle transformation matrix based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device includes: performing point cloud registration processing on the edge image point cloud and the baffle laser point cloud according to a preset distance iteration algorithm to obtain a point cloud transformation matrix between the baffle laser point cloud and the edge image point cloud; and determining the bird's-eye view vehicle transformation matrix based on the extrinsic parameters of the laser device and the point cloud transformation matrix.

[0013] In one embodiment, determining the target bird's-eye view transformation matrix based on the obtained bird's-eye view viewable range, a preset scaling ratio, and the bird's-eye view vehicle body transformation matrix includes: converting the identification code corner point to the vehicle body coordinate system where the mobile device is located according to the bird's-eye view vehicle body transformation matrix to obtain the current coordinates of the identification code corner point in the vehicle body coordinate system; converting the current coordinates according to the bird's-eye view viewable range and the preset scaling ratio to obtain bird's-eye view coordinates; and performing calibration processing based on the bird's-eye view coordinates and the pixel coordinates of the identification code corner point to obtain the target bird's-eye view transformation matrix.

[0014] A second aspect of this application provides a camera-based bird's-eye view calibration device, which is applied to a mobile device. The mobile device includes the camera and a pre-calibrated laser device. The device includes: an acquisition module, configured to acquire a calibration image obtained by the camera capturing an image of a preset calibration checkerboard and a laser point cloud of a pre-calibrated baffle acquired by the laser device; the calibration checkerboard includes an identification code, and the preset baffle is disposed on the edge of the calibration checkerboard and perpendicular to the plane where the calibration checkerboard is located; and a first calibration module, configured to calibrate the image based on the pixel coordinates of the identification code in the calibration image and the pre-calibrated laser point cloud of the pre-calibrated baffle. The identification code is processed by bird's-eye view calibration based on its real coordinates in the calibration checkerboard to obtain an initial bird's-eye view transformation matrix; the edge determination module is used to obtain an initial bird's-eye view of the calibration checkerboard based on the initial bird's-eye view transformation matrix, and to obtain the edge image point cloud of the edges in the initial bird's-eye view; the matrix determination module is used to determine the bird's-eye view vehicle transformation matrix based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device; the second calibration module is used to determine the target bird's-eye view transformation matrix based on the visible range of the obtained bird's-eye view, the preset scaling ratio, and the bird's-eye view vehicle transformation matrix.

[0015] A third aspect of this application provides an electronic device including a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the aforementioned camera-based bird's-eye view mapping method.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described camera-based bird's-eye view mapping method.

[0017] The above scheme is applied to a mobile device, which includes a camera and a pre-calibrated laser device. The calibration scene is pre-set with a calibration checkerboard and a baffle. The calibration checkerboard includes an identification code, and the pre-set baffle is positioned at the edge of the calibration checkerboard and perpendicular to the plane of the checkerboard. By acquiring a calibration image of the pre-set calibration checkerboard using the camera and identifying the identification code, bird's-eye view calibration processing is performed based on the pixel coordinates of the identification code in the calibration image and its actual coordinates in the calibration checkerboard, resulting in an initial bird's-eye view transformation matrix. Then, an initial bird's-eye view of the calibration checkerboard can be obtained based on the initial bird's-eye view transformation matrix, along with edge image point clouds of the edges in the initial bird's-eye view. After acquiring the baffle laser point cloud acquired by the laser device from the pre-set baffle, the bird's-eye view vehicle transformation matrix can be determined based on the edge image point cloud, the baffle laser point cloud, and the pre-calibrated laser extrinsic parameters of the laser device. Then, based on the visible range of the acquired bird's-eye view, the preset zoom ratio, and the bird's-eye view vehicle transformation matrix, the target bird's-eye view transformation matrix of the camera is determined with the vehicle coordinate system as the center reference. Therefore, compared with the traditional calibration process, this method, which uses laser data and image data for bird's-eye view calibration, does not require strict requirements on the parking position of the mobile device, thus improving calibration efficiency.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic flowchart of an exemplary embodiment of the camera-based bird's-eye view mapping method of this application; Figure 2 This is an exemplary calibration scene diagram in the camera-based bird's-eye view calibration method of this application; Figure 3 This is an exemplary specific scene diagram of the camera-based bird's-eye view mapping method of this application; Figure 4 This is a block diagram illustrating a camera-based bird's-eye view positioning device in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0021] The solution of this embodiment will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] For ease of understanding, one of the applicable scenarios of this application is illustrated by example.

[0025] Camera calibration is a technique that calculates and determines the parameters of the camera's imaging geometric model. It is used to establish the correspondence between three-dimensional space and two-dimensional image coordinates and is a key step in three-dimensional modeling, image measurement, and machine vision.

[0026] For example, in the driving scenarios of mobile devices, taking mobile robots as an example, a bird's-eye view (BEV) is a commonly used method for representing their environment. A BEV scene can be constructed based on camera data from one or more viewpoints, which involves the accurate calibration of camera extrinsic parameters.

[0027] However, existing calibration methods typically require designing a large calibration cloth that surrounds the mobile device, based on its actual dimensions. Furthermore, these methods may fail to function correctly due to partial obstruction, changes in lighting, or tilted viewing angles. Additionally, the calibration process necessitates precise positioning of the mobile device at the calibration location (e.g., the center lines of the front and rear checkerboard grids on the vehicle body must align with the horizontal center line of the vehicle's placement frame, and the center lines of the left and right checkerboard grids on the vehicle body must be aligned as closely as possible with the lines connecting the left and right rearview mirrors), which is extremely inconvenient and makes it difficult to handle rapid recalibration of the mobile device after camera replacement or mechanical structure adjustments.

[0028] Please see Figure 1 , Figure 1This is a flowchart illustrating an exemplary embodiment of the camera-based bird's-eye view calibration method of this application. The method is applied to mobile devices (such as autonomous mobile robots (AMRs), automobiles, etc., which are not limited here), and the mobile device is equipped with at least one camera and a pre-calibrated laser device.

[0029] For ease of understanding and explanation, the mobile device described in this application is mainly based on mobile robots such as AMRs.

[0030] For example, a mobile robot may include at least a mobile chassis (including a motion controller, motor, battery, embedded computer, etc.), and the mobile robot may also have three color cameras with different perspectives (which may have overlapping views) installed at a certain height above the ground (for example, they may be installed on the front, left and right sides of the vehicle body respectively), as well as a 2D ranging laser device.

[0031] In specific calibration scenarios, a calibration checkerboard (or checkerboard calibration cloth, which can be set with reference to the style of ChArUco calibration cloth in this technical field) is pre-set. It can include multiple checkerboard squares (black checkerboard squares and white checkerboard squares), and the white checkerboard squares can have ArUco identification codes, which will not be elaborated here. Preferably, in order to ensure that the calibration cloth can be observed by the front, left and right three-view cameras on the robot, and to facilitate carrying, the calibration cloth can be set as a long and narrow rectangular shape (width > height).

[0032] For example, before formally implementing the calibration method of this application, the actual length, width and other dimensions of the calibration checkerboard grid, as well as the number of rows and columns of the checkerboard grid, are preset. This allows the automatic generation of the ChArUco calibration cloth. Compared with traditional calibration cloths, ChArUco can locate the corresponding corner point position based on the code value of each identification code, thereby solving the problem of insufficient overlap of the camera's field of view.

[0033] For reference, please see below. Figure 2 As shown, Figure 2 This is an exemplary calibration scene diagram of the camera-based bird's-eye view calibration method of this application. Preset baffles are also provided at the upper left and upper right edges of the calibration checkerboard. The preset baffles are perpendicular to the plane where the calibration checkerboard is located, and reflective stickers (used to provide point clouds from the overhead view to the laser device) can be attached to the baffles at each edge and surround the edge corner points (such as P1 and Pr) of the calibration checkerboard.

[0034] It should be noted that the specific arrangement of the baffles can be adjusted as needed according to the actual application scenario. For example, two baffles can be set at each edge corner to form the L-shape shown in the attached figure, providing an L-shaped point cloud; or other numbers of baffles that can achieve the same basic effect can also be set, which is not limited here.

[0035] For the sake of illustration, Figure 2 The identification codes in the example checkerboard grid are represented in numerical form. In practical applications, the codes can be set according to the ArUco code style in this technical field, which will not be elaborated further.

[0036] Therefore, when formally implementing the calibration method of this application, the calibration checkerboard and baffle can be placed on the ground in front of the robot (i.e., in the image acquisition direction of the robot's cameras), so that cameras from different perspectives can completely observe at least one ArUco code. Then, based on the corner points of the ArUco code identified by cameras from different perspectives, the initial IPM (Inverse Perspective Mapping) matrix can be calibrated.

[0037] Then, the extrinsic parameters of the bird's-eye view can be calibrated based on the baffle contour data collected by 2D laser and the corresponding pixel positions of the BEV (bird's-eye view), and the target IPM matrix finally projected onto the center of the robot's body can be determined, thus realizing the calibration of the entire process through a small calibration cloth.

[0038] Compared to traditional IPM calibration methods that require large calibration cloths to obtain accurate poses, leading to inconvenience in carrying them, this application proposes a calibration method based on ChArUco calibration cloth and 2D laser, which can determine the transformed extrinsic parameters by aligning the two observation data. Therefore, the calibration cloth only needs to retain the size of the camera recognition area, significantly reducing the size of traditional calibration cloths. Furthermore, using ChArUco as the identification code also improves the robustness and accuracy of recognition, resulting in an overall improvement in calibration efficiency.

[0039] Specifically, the calibration method of this application may include the following steps: Step S110: Obtain the calibration image obtained by the camera capturing the image of the preset calibration checkerboard and the laser point cloud of the laser device capturing the image of the preset baffle; the calibration checkerboard includes an identification code, and the preset baffle is set on the edge of the calibration checkerboard and perpendicular to the plane where the calibration checkerboard is located.

[0040] For example, each camera has its intrinsic parameters pre-calibrated, so distortion correction can be performed on the calibration images (also known as camera images) acquired from their respective viewpoints based on their intrinsic parameters, eliminating geometric distortions introduced by the optical characteristics of the camera lens, and making the acquired images conform to the ideal perspective projection model.

[0041] Alternatively, the laser point cloud of the baffle can be obtained by collecting the laser light emitted by the 2D laser device hitting the preset baffle.

[0042] It should be noted that, optionally, the step of obtaining the baffle laser point cloud can be performed in step S110, or it can be adjusted to other steps (before the baffle laser point cloud is needed) as needed, and there is no limitation here.

[0043] The method for setting the calibration chessboard grid and its preset baffles can be referred to the aforementioned example, and will not be repeated here.

[0044] Step S120: Perform bird's-eye view calibration processing based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard to obtain the initial bird's-eye view transformation matrix.

[0045] It should be noted that after each camera acquires a calibration image, the calibration image can be identified by a preset recognition algorithm to obtain the identification code in the calibration image and the pixel coordinates of the identification code in the calibration image (coordinates in the image coordinate system). Furthermore, the actual coordinates of the identification code in the calibration checkerboard (coordinates in the world coordinate system) can be obtained based on the code value of the identification code.

[0046] The coordinates of the identifier code can refer to the coordinates of the corner points of the identifier code (referred to as the corner coordinates of the identifier code), or the coordinates calculated further based on the corner coordinates of the identifier code (such as the coordinates of the center point of the identifier code), without any limitation here.

[0047] It is understandable that the principle of calculating the IPM matrix is ​​to establish a mapping relationship between the image plane (image coordinate system) and the BEV viewpoint plane through homography transformation.

[0048] For example, its mathematical expression can be: Formula 1 in, The coordinates of the BEV image (initial bird's-eye view) The coordinates of the camera image (calibration image), at the same corner point. and They can form corresponding point pairs.

[0049] Therefore, the pixel coordinates of the same corner point in the calibration image and its coordinates in the initial bird's-eye view image can be used to form a point pair. The bird's-eye view coordinates can be calculated from the true coordinates of the corner point within the calibration checkerboard. During IPM matrix calibration, the matrix has nine variables, but homogeneous coordinates are scale-invariant, and are typically... Set to 1. In addition, each pair of points can provide two constraints, so at least four corner points (corresponding to four pairs of points) can be selected from the multiple identified identifiers to provide eight constraints. This allows the determination of the remaining eight variables of the IPM matrix, thus completing the bird's-eye view calibration process and obtaining the initial bird's-eye view transformation matrix.

[0050] Step S130: Obtain the initial bird's-eye view of the calibration checkerboard grid based on the initial bird's-eye view transformation matrix, and obtain the edge image point cloud of the edges in the initial bird's-eye view.

[0051] The initial bird's-eye view refers to the image of the checkerboard pattern from a bird's-eye view perspective.

[0052] The edge refers to the edge of the checkerboard.

[0053] Edge image point cloud refers to point cloud data determined from the edges of the labeled checkerboard in the initial bird's-eye view.

[0054] Based on the steps described above, after obtaining the initial bird's-eye view transformation matrix, the collected calibration chessboard image can be processed by inverse perspective transformation to obtain the initial bird's-eye view of the calibration chessboard from a bird's-eye view perspective.

[0055] Furthermore, a preset recognition algorithm can be used to identify the checkerboard pattern in the initial bird's-eye view, determine the edges of the checkerboard pattern in the initial bird's-eye view, and obtain the corresponding edge image point cloud.

[0056] The method for obtaining edge image point clouds may include, but is not limited to, randomly obtaining point cloud data from the edges of the checkerboard pattern marked in the initial bird's-eye view, or obtaining point cloud data according to a certain point cloud selection strategy; no limitation is made here.

[0057] Step S140: Determine the bird's-eye view vehicle transformation matrix based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device.

[0058] The bird's-eye view vehicle transformation matrix refers to the transformation matrix between the real coordinate system centered on the initial bird's-eye view and the vehicle coordinate system where the mobile device is located.

[0059] In conjunction with the foregoing embodiments, the step of obtaining the baffle laser point cloud can be performed in step S110 or after step S130, etc., and there is no limitation here.

[0060] according to Figure 2 As shown, since the preset baffle is set at the edge of the calibration checkerboard, the edge image point cloud of the calibration checkerboard determined from the initial bird's-eye view and the baffle laser point cloud obtained from the laser device both belong to the edge point cloud of the calibration checkerboard, and the edge image point cloud and the baffle laser point cloud should match.

[0061] Therefore, by referring to the IPM calibration principle of the aforementioned embodiment, the transformation matrix between the laser coordinate system and the real coordinate system centered on the initial bird's-eye view can be obtained by matching the edge image point cloud and the baffle laser point cloud (for ease of distinction and explanation, it can also be called the point cloud transformation matrix).

[0062] Then, based on the pre-calibrated laser extrinsic parameters of the laser device (the mapping relationship between the laser and the vehicle coordinate system), the point cloud transformation matrix can be further transformed to obtain the transformation matrix between the real coordinate system centered on the initial bird's-eye view and the vehicle coordinate system (i.e., the bird's-eye view vehicle transformation matrix).

[0063] Step S150: Determine the target bird's-eye view transformation matrix based on the obtained bird's-eye view visible range, preset scaling ratio, and bird's-eye view vehicle transformation matrix.

[0064] Referring to the previous example, after obtaining the bird's-eye view vehicle transformation matrix, the origin of the vehicle coordinate system can be used as the reference point to transform the corner coordinates of the identified identifier in the calibration image to the vehicle coordinate system using the bird's-eye view vehicle transformation matrix. Then, it is necessary to obtain the preset viewing range (visible range of the bird's-eye view) and preset scaling ratio of the bird's-eye view. Based on the visible range of the bird's-eye view and the preset scaling ratio, the camera intrinsic parameters of the BEV view (which can convert the real coordinates to coordinates under the BEV view) are determined, and the pixel coordinates of the corresponding points under the BEV view are calculated (for example, based on the corner coordinates in the identified calibration image and the corresponding pixel coordinates in the bird's-eye view centered on the vehicle body).

[0065] Therefore, by referring to the IPM calibration principle in the aforementioned embodiments, the IPM matrix can be recalibrated based on the corner coordinates of the identified identifier in the calibration image and the pixel coordinates of the same corner that have been transformed to the BEV viewpoint, thus obtaining the target transformation matrix.

[0066] As can be seen, this application applies to a mobile device, which includes a camera and a pre-calibrated laser device. The calibration scene is pre-set with a calibration checkerboard and a baffle. The calibration checkerboard includes an identification code, and the pre-set baffle is positioned at the edge of the calibration checkerboard and perpendicular to the plane of the checkerboard. By acquiring a calibration image of the pre-set calibration checkerboard using the camera and identifying the identification code, bird's-eye view calibration processing can be performed based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard, resulting in an initial bird's-eye view transformation matrix. Then, an initial bird's-eye view of the calibration checkerboard can be obtained based on the initial bird's-eye view transformation matrix, and edge image point clouds of the edges in the initial bird's-eye view can be acquired. After acquiring the baffle laser point cloud acquired by the laser device from the pre-set baffle, the bird's-eye view vehicle transformation matrix can be determined based on the edge image point cloud, the baffle laser point cloud, and the pre-calibrated laser extrinsic parameters of the laser device. Furthermore, the target transformation matrix of the camera is calibrated with the vehicle coordinate system as the central reference based on the initial bird's-eye view transformation matrix and the bird's-eye view vehicle transformation matrix. Therefore, compared with the traditional calibration process, this method, which uses aerial view calibration by collecting laser data and image data, does not require strict requirements on the parking location of mobile devices, thus improving calibration efficiency.

[0067] Based on the above embodiments, this embodiment describes the method prior to step S120.

[0068] Specifically, in this embodiment, the method before performing bird's-eye view calibration processing based on the pixel coordinates of the identifier in the calibration image and the actual coordinates of the identifier in the calibration checkerboard to obtain the initial bird's-eye view transformation matrix includes at least the following steps S210 to S220: Step S210: Identify the corner points of the identification code in the calibration image and obtain the pixel coordinates of the corner points.

[0069] Step S220: Determine the true coordinates of the corner point of the identification code in the calibration checkerboard based on the pixel coordinates, the true scale between the calibration image and the calibration checkerboard, and the checkerboard size information.

[0070] For example, when determining the initial IPM matrix, a point in the calibration cloth can be used as the reference coordinate (e.g., the point at the center of the lower edge of the calibration cloth is selected as the reference coordinate). Therefore, the true coordinates of the corner points of the ArUco identifier under the reference coordinate can be calculated based on the calibration cloth template image and the actual size of the calibration cloth. ].

[0071] For example, the scale ratio s (real scale) between the calibration fabric template image used in making the calibration fabric and its actual physical space is known and obtainable. Using a pre-defined recognition algorithm, the pixel coordinates of the corner points of each identification code in the calibration image can be identified. .

[0072] Therefore, in a scenario where the bottom center point of the calibration checkerboard is used as the reference coordinate, the height and width of the calibration checkerboard template image (such as...) can be combined. and The checkerboard grid size information is calculated using a scale ratio 's', and then the corresponding real coordinates are calculated by combining the pixel coordinates of each corner point. Its mathematical expression can be: Formula 2 Based on the above embodiments, this embodiment describes the method in step S120.

[0073] Specifically, in this embodiment, the method for obtaining an initial bird's-eye view transformation matrix by performing bird's-eye view calibration processing based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard grid includes at least the following steps S310 to S330: Step S310: Obtain the preset zoom level and the visible range of the bird's-eye view.

[0074] Step S320: Convert the real coordinates into bird's-eye view coordinates according to the zoom level and the visible range of the bird's-eye view.

[0075] Step S330: Determine the initial bird's-eye view transformation matrix based on the pixel coordinates of the identification code corner point and the bird's-eye view image coordinates.

[0076] For reference, Formula 3 can be used to determine the preset scaling ratio. This refers to the scaling phenomenon that occurs when transforming real coordinates to BEV image pixels. This scaling ratio can be set and obtained in advance. Additionally, the visible frontal area of ​​the bird's-eye view can also be set and obtained in advance. and the visible range on the left .

[0077] The intrinsic parameter matrix between the real coordinates and the pixel coordinates from the BEV viewpoint can be determined based on the scaling ratio and the visible range of the bird's-eye view. . This represents the camera intrinsics of the BEV viewpoint. In the example case, the BEV viewpoint height is set to 1m, which is equivalent to the transformation matrix between real-world coordinates in the BEV viewpoint and BEV image coordinates (pixel coordinates in the BEV viewpoint).

[0078] Therefore, the mathematical expression for converting real coordinates to bird's-eye view coordinates can be: Formula 3 In other words, by using the aforementioned example method, the real coordinates corresponding to each pixel coordinate can be transformed into bird's-eye view coordinates, thus determining the bird's-eye view coordinates corresponding to the pixel coordinates. In turn, the initial IPM matrix Hinit (initial bird's-eye view transformation matrix) can be determined based on the point pairs formed by the pixel coordinates of multiple corner points and the bird's-eye view coordinates.

[0079] Specifically, each ArUco code has a unique code value and the corner points are stored in the same order. A preset recognition algorithm is used to identify the pixel coordinates of the code corner points in the calibration images acquired from different viewpoints. Pixel coordinates can be obtained using the methods described in the examples above. coordinates of bird's-eye view image The correspondence.

[0080] Therefore, the IPM matrix can be obtained using Formula 1. In practical applications, each camera viewpoint can detect multiple identifiers, and each identifier can provide four corner points, meaning that a viewpoint can provide more than four pairs of points. Therefore, the initial bird's-eye view transformation matrix (initial IPM matrix) can be calculated based on each pair of points.

[0081] Based on the above embodiments, this embodiment describes the method in step S330.

[0082] Specifically, in this embodiment, the method for determining the initial bird's-eye view transformation matrix based on the pixel coordinates of the identification code corner point and the bird's-eye view image coordinates includes at least the following steps S410 to S440: Step S410: Obtain the corner points of the multiple identification codes.

[0083] Step S420: Select multiple target identifier corner points from multiple identifier corner points according to a preset random sampling consensus algorithm.

[0084] Step S430: Based on the pixel coordinates of multiple target identification code corner points and the bird's-eye view image coordinates, determine multiple initial transformation matrices and the reprojection error of each initial transformation matrix.

[0085] Step S440: Determine the initial bird's-eye view transformation matrix from multiple initial transformation matrices based on the reprojection error.

[0086] As described in conjunction with the foregoing embodiments, multiple identification code corner points can be identified using a multi-view camera, and the corresponding point pair for each identification code corner point can be determined by referring to the foregoing example method.

[0087] Then, a random sampling consensus algorithm (such as the RANSAC algorithm) can be used for each identifier corner point to randomly select 4 pairs of points and repeat the calculation of the candidate IPM matrix and the matrix reprojection error corresponding to each candidate IPM matrix N times (N is a positive integer preset as needed).

[0088] Furthermore, based on the matrix reprojection error corresponding to each candidate IPM matrix, the matrix reprojection error can be determined from each candidate IPM matrix. The matrix (initial IPM matrix, also known as the initial bird's-eye view transformation matrix). The matrix with the smallest reprojection error can be selected from these matrices. The matrix is ​​the initial IPM matrix, and there are no restrictions here.

[0089] Based on the above embodiments, this embodiment describes the method in step S130.

[0090] Specifically, in this embodiment, the method for obtaining an initial bird's-eye view of the calibration checkerboard based on an initial bird's-eye view transformation matrix, and for obtaining edge image point clouds of the edges in the initial bird's-eye view, includes at least the following steps S510 to S540: Step S510: Transform the calibration image according to the initial bird's-eye view transformation matrix to obtain the initial bird's-eye view.

[0091] Step S520: Identify the specified corner points in the initial bird's-eye view.

[0092] Step S530: Determine the edge corner points of the marked checkerboard in the initial bird's-eye view based on the checkerboard size information and the specified corner points.

[0093] Step S540: Select line segment points based on the baffle size information, edge corner points and preset distances to obtain the edge image point cloud.

[0094] Referring to the foregoing embodiments, after obtaining the initial bird's-eye view transformation matrix, the calibration image (camera image) can be projected onto the image under the bird's-eye view centered on the coordinate system defined by the calibration cloth, thus obtaining the initial bird's-eye view.

[0095] It should be noted that in real-world application scenarios, when the mobile device is running, a bird's-eye view centered on the vehicle body is required. Therefore, this application can recalibrate the target IPM matrix using subsequent methods to conform to the actual application scenario.

[0096] The specified corner point can be set as needed in the actual application scenario. In this embodiment, it can be... Figure 3 Taking the application scenario shown as an example, Figure 3 This is an exemplary scene diagram of the camera-based bird's-eye view mapping method of this application, in which the identified mapping checkerboard grid can be selected. , , , The corner points are designated as corner points (for easy identification and calculation).

[0097] Furthermore, the size information of the chessboard grid (such as the actual width and height of each square) can be used to define the chessboard grid. and The coordinates of the edge corner points of the checkerboard grid in the initial bird's-eye view are determined using the coordinates of the specified corner points. Here, edge corner points refer to corner points located on the edge of the checkerboard grid, such as... , , , .

[0098] Specifically, the center coordinates of the chessboard can be calculated first based on the coordinates of the specified corner points. Then, based on the difference between the x and y coordinates of each specified corner point, the width and height distances from the edge corner points to the center coordinates can be determined. and Then determine the offset angle of the calibration cloth. Therefore, the coordinates of the edge corner points can be determined based on the horizontal and vertical coordinates of the center coordinates, the width and height distances from the edge corner points to the center coordinates, and the offset angle of the calibration cloth.

[0099] For example, its mathematical expression can be:

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] Furthermore, the size information of the preset baffle is known and obtainable, such as... , , , Therefore, by combining the baffle size information and the position of the edge corner points, the position of the baffle edge points can be determined as follows: , , , Therefore, we can start from line segments. , , , Line segment points are selected at preset distances to obtain the edge image point cloud.

[0107] Based on the above embodiments, this embodiment describes the method in step S540.

[0108] Specifically, in this embodiment, the method for obtaining an edge image point cloud by selecting line segment points based on the baffle size information, edge corner points, and preset distances includes at least the following steps S610 to S630: Step S610: Transform the coordinates of the edge corner points according to the initial bird's-eye view transformation matrix to obtain the true edge corner points.

[0109] Step S620: Determine the edge points of the baffle based on the preset linear interpolation algorithm, the actual edge corner points, and the baffle size information.

[0110] Step S630: Select line segment points from the line segments between the edge points of the baffle and the corner points of the real edge according to the preset distance to obtain the edge image point cloud.

[0111] In conjunction with the aforementioned embodiments, during the selection of edge image point clouds, it is necessary to transform the coordinates of edge corner points in the calibration image by inverting the intrinsic parameter matrix of the bird's-eye view to obtain the true edge corner points under the BEV perspective, and then determine the edge points by combining the actual size of the baffle.

[0112] For example, the coordinates of edge corner points in a camera image can be obtained according to Formula 3. , , , Transformation to real-scale coordinates , , , Then, the edge points of the baffle can be obtained using a preset linear interpolation algorithm. ) to determine the coordinates of the reference coordinate system in which the calibration cloth is located ( This leads to the line segment. , can every By identifying the line segment points in each line segment, the actual point cloud corresponding to the multi-view image is obtained. (i.e., edge image point cloud).

[0113] Using one of the baffle edge points Taking the calculation process as an example, its mathematical expression can be:

[0114]

[0115]

[0116]

[0117]

[0118] Based on the above embodiments, this embodiment describes the method in step S140.

[0119] Specifically, in this embodiment, the method for determining the bird's-eye view vehicle transformation matrix based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device includes at least the following steps S710 to S720: Step S710: Perform point cloud registration processing on the edge image point cloud and the baffle laser point cloud according to the preset distance iteration algorithm to obtain the point cloud transformation matrix between the baffle laser point cloud and the edge image point cloud.

[0120] Step S720: Determine the bird's-eye view vehicle transformation matrix based on the extrinsic parameters of the laser device and the point cloud transformation matrix.

[0121] The edge image point cloud will be described in conjunction with the foregoing embodiments. It is the camera's perception of the edges of the calibration checkerboard, and the baffle laser point cloud. It is the perception of the edge (preset baffle) of the calibrated checkerboard by a 2D laser device.

[0122] Therefore, after obtaining the edge image point cloud and the baffle laser point cloud, the extrinsic transformation matrix between the laser and the initial BEV coordinate system in the aforementioned example can be determined using a preset distance iteration algorithm (such as the ICP iterative nearest point algorithm). (i.e., point cloud transformation matrix).

[0123] Furthermore, the laser extrinsic parameters pre-calibrated by the laser device are obtained. and extrinsic transformation matrix Then the transformation matrix between the initial BEV coordinate system and the vehicle body can be determined. (i.e., the bird's-eye view vehicle transformation matrix).

[0124] For example, its mathematical expression can be:

[0125] Based on the above embodiments, this embodiment describes the method in step S150.

[0126] Specifically, in this embodiment, the method for determining the target bird's-eye view transformation matrix based on the obtained bird's-eye view view's visible range, a preset scaling ratio, and a bird's-eye view vehicle transformation matrix includes at least the following steps S810 to S830: Step S810: Based on the bird's-eye view vehicle body transformation matrix, the corner point of the identification code is transformed to the vehicle body coordinate system where the mobile device is located, and the current coordinates of the corner point of the identification code in the vehicle body coordinate system are obtained.

[0127] Step S820: Convert the current coordinates according to the visible range of the bird's-eye view and the preset zoom ratio to obtain the bird's-eye view coordinates.

[0128] Step S830: Calibration processing is performed based on the bird's-eye view coordinates and the pixel coordinates of the identification code corner points to obtain the target bird's-eye view transformation matrix.

[0129] To explain in conjunction with the steps above, since in actual application scenarios, when the mobile device is running, it is necessary to use a bird's-eye view centered on the mobile device body.

[0130] Therefore, in determining the BEV perspective and the bird's-eye view transformation matrix of the vehicle body... Then, the process of recalibrating the IPM matrix can be entered to obtain the target IPM matrix (i.e., the target bird's-eye view transformation matrix of the bird's-eye view when each camera is centered on the vehicle body).

[0131] For example, during the recalibration process, it is necessary to select the origin of the vehicle coordinate system as the reference point.

[0132] Based on the initial bird's-eye view transformation matrix, the BEV intrinsic parameter matrix, and the bird's-eye view vehicle body transformation matrix, the corner points of the identification codes are transformed to the vehicle body coordinate system where the mobile device is located, obtaining the current coordinates of the corner points in the vehicle body coordinate system. Then, the current coordinates are transformed again according to the BEV intrinsic parameter matrix (determined by the visible range of the bird's-eye view and the preset scaling ratio) to obtain the bird's-eye view coordinates centered on the vehicle body. Subsequently, based on the pixel coordinates of the identified corner points of the identification codes and the corresponding bird's-eye view coordinates centered on the vehicle body, the target bird's-eye view transformation matrix of each camera's bird's-eye view image centered on the vehicle body can be determined.

[0133] In some specific implementations, the above method can first use the center of the calibration cloth on the lower side as the reference coordinate. Similarly, according to the aforementioned example method, the corner points (pixel coordinates) of the identification code in the calibration image are identified. The pixel coordinates of the identification code corner points are transformed to the pixel coordinates of the initial bird's-eye view using the initial bird's-eye view transformation matrix. Then, the current coordinates are transformed to the real coordinate system of the initial bird's-eye view using the inverse matrix of the bird's-eye view intrinsic parameters. Finally, it is multiplied by the bird's-eye view vehicle body transformation matrix. Transform to the vehicle coordinate system to obtain the current coordinates of the identification code corner point in the vehicle coordinate system.

[0134] The calibration image used in this step may be the same as or different from the calibration image used in step S110 (for example, it may be a camera image obtained by re-acquiring the calibration checkerboard pattern), and there is no limitation here.

[0135] Furthermore, the bird's-eye view intrinsic parameter matrix is ​​determined based on the visible range of the obtained bird's-eye view and the preset scaling ratio. Then, the current coordinates are transformed through the bird's-eye view intrinsic parameter matrix, thereby obtaining the pixel coordinates (i.e., bird's-eye view coordinates) from the BEV perspective centered on the vehicle coordinate system.

[0136] It should be noted that the bird's-eye view intrinsic parameter matrix in this embodiment can be determined during the recalibration process. Alternatively, it can be determined when calibrating the initial bird's-eye view transformation matrix in step S120 (or in step 320), and can be directly obtained and called in step S830 (or step S150). No limitation is made here.

[0137] For details, please refer to the bird's-eye view intrinsic parameter matrix in the aforementioned embodiments. The explanation is omitted here.

[0138] Therefore, after obtaining the point pairs with the same corner points (the pixel coordinates of the same corner points in the calibration image and the bird's-eye view coordinates from the BEV perspective), the final target IPM matrix can be calculated by referring to the IPM matrix calibration principle in the aforementioned embodiments. (i.e., the target bird's-eye view transformation matrix of each camera with the vehicle body as the center).

[0139] Therefore, the calibration method provided in this application can be used for calibration based on the long and narrow form of the ChArUco calibration cloth. The uniqueness of the ChArUco corner markers improves matching accuracy and enhances robustness in different environments, requiring no manual intervention. The long and narrow design ensures both a shared viewing area from multiple perspectives and portability of the calibration cloth.

[0140] Furthermore, by introducing 2D laser data, a coordinate system centered at any coordinate can be flexibly calibrated to an external parameter centered on the vehicle body. For example, in the example of this application, by observing the same area (calibrating the edges and baffles of the checkerboard) with both a camera and a laser, and matching the laser point cloud data with the image point cloud data from a bird's-eye view, a target IPM matrix centered on the vehicle body can be obtained, thus improving calibration efficiency.

[0141] It should be further noted that the execution entity of the camera-based bird's-eye view mapping method can be a camera-based bird's-eye view mapping device. For example, the camera-based bird's-eye view mapping method can be executed by a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the camera-based bird's-eye view mapping method can be implemented by a processor calling computer-readable instructions stored in memory.

[0142] Figure 4 This is a block diagram illustrating a camera-based bird's-eye view calibration device, as shown in an exemplary embodiment of this application. The device can be applied to a mobile device, which includes a camera and a pre-calibrated laser device. A calibration checkerboard and a baffle are pre-set in the calibration scene. The calibration checkerboard includes an identification code, and the pre-set baffle is disposed at the edge of the calibration checkerboard and perpendicular to the plane of the calibration checkerboard. Figure 4 As shown, the exemplary camera-based bird's-eye view calibration device 400 includes: an acquisition module 410, a first calibration module 420, an edge determination module 430, a matrix determination module 440, and a second calibration module 450. Specifically: The acquisition module 410 is used to acquire the calibration image obtained by the camera capturing the image of the preset calibration checkerboard and the laser point cloud of the laser device capturing the image of the preset baffle. The calibration checkerboard includes an identification code, and the preset baffle is set on the edge of the calibration checkerboard and perpendicular to the plane where the calibration checkerboard is located.

[0143] The first calibration module 420 is used to perform bird's-eye view calibration processing based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard to obtain the initial bird's-eye view transformation matrix.

[0144] The edge determination module 430 is used to obtain the initial bird's-eye view of the calibration checkerboard based on the initial bird's-eye view transformation matrix, and to obtain the edge image point cloud of the edges in the initial bird's-eye view.

[0145] The matrix determination module 440 is used to determine the bird's-eye view vehicle body transformation matrix based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device.

[0146] The second calibration module 450 is used to determine the target bird's-eye view transformation matrix based on the visible range of the acquired bird's-eye view, the preset scaling ratio, and the bird's-eye view vehicle transformation matrix.

[0147] In this exemplary camera-based bird's-eye view calibration device, a calibration image is acquired by the camera capturing images of a preset calibration checkerboard, and the identification code within it is identified. Bird's-eye view calibration processing is then performed based on the pixel coordinates of the identification code in the calibration image and its actual coordinates within the calibration checkerboard, resulting in an initial bird's-eye view transformation matrix. Then, an initial bird's-eye view of the calibration checkerboard is obtained based on the initial bird's-eye view transformation matrix, and edge image point clouds of the edges in the initial bird's-eye view are acquired. After acquiring the laser point cloud of the preset baffle acquired by the laser device, the bird's-eye view vehicle transformation matrix is ​​determined based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device. Furthermore, based on the acquired bird's-eye view view's visible range, the preset scaling ratio, and the bird's-eye view vehicle transformation matrix, the target transformation matrix of the camera is calibrated with the vehicle coordinate system as the central reference. Therefore, compared to traditional calibration processes, this method, which uses laser data and image data for bird's-eye view calibration, does not require strict adherence to the parking position of the mobile device, thus improving calibration efficiency.

[0148] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0149] The functions of each module can be found in the embodiment of the camera-based bird's-eye view mapping method, and will not be repeated here.

[0150] Please see Figure 5 , Figure 5This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-described embodiments of the camera-based bird's-eye view mapping method. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0151] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described camera-based bird's-eye view mapping method embodiments. Processor 102 may also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 102 may be implemented using integrated circuit chips.

[0152] In this exemplary electronic device, a calibration image is acquired by a camera capturing images of a preset calibration checkerboard, and the identification code within it is identified. Bird's-eye view calibration is then performed based on the pixel coordinates of the identification code in the calibration image and its actual coordinates within the calibration checkerboard, yielding an initial bird's-eye view transformation matrix. Then, an initial bird's-eye view of the calibration checkerboard is obtained based on this initial transformation matrix, along with edge image point clouds of the edges within the initial bird's-eye view. After acquiring the laser point cloud of the preset baffle captured by the laser device, the bird's-eye view vehicle transformation matrix is ​​determined based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device. Furthermore, based on the acquired bird's-eye view's visible range, a preset scaling ratio, and the bird's-eye view vehicle transformation matrix, the target transformation matrix of the camera is calibrated with the vehicle coordinate system as the central reference. Therefore, compared to traditional calibration processes, this method, which uses laser data and image data for bird's-eye view calibration, eliminates the need for strict requirements on the parking position of the mobile device, thus improving calibration efficiency.

[0153] Please see Figure 6 , Figure 6This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above-described embodiments of the camera-based bird's-eye view mapping method.

[0154] In this exemplary storage medium, by running the program instructions in the storage medium, a calibration image obtained by the camera capturing images of a preset calibration checkerboard is acquired, and the identification code in the image is identified. Bird's-eye view calibration processing is performed based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard, resulting in an initial bird's-eye view transformation matrix. Then, an initial bird's-eye view of the calibration checkerboard can be obtained based on the initial bird's-eye view transformation matrix, and edge image point clouds of the edges in the initial bird's-eye view are acquired. After acquiring the laser point cloud of the preset baffle acquired by the laser device, the bird's-eye view vehicle transformation matrix can be determined based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device. Furthermore, based on the acquired bird's-eye view view's visible range, the preset scaling ratio, and the bird's-eye view vehicle transformation matrix, the target transformation matrix of the camera is calibrated with the vehicle coordinate system as the central reference. Therefore, compared to the traditional calibration process, this method, which uses laser data and image data for bird's-eye view calibration, does not require strict adherence to the parking position of the mobile device, thus improving calibration efficiency.

[0155] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0156] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A camera-based bird's-eye view mapping method, characterized in that, The method is applied to a mobile device, the mobile device including the camera and a pre-calibrated laser device, the method comprising: The system acquires a calibration image obtained by the camera capturing images of a preset calibration checkerboard and a laser point cloud of the laser device capturing images of a preset baffle. The calibration checkerboard includes an identification code, and the preset baffle is disposed on the edge of the calibration checkerboard and perpendicular to the plane in which the calibration checkerboard is located. Bird's-eye view calibration is performed based on the pixel coordinates of the identification code in the calibration image and the actual coordinates of the identification code in the calibration checkerboard to obtain the initial bird's-eye view transformation matrix. The initial bird's-eye view of the calibration chessboard is obtained based on the initial bird's-eye view transformation matrix, and the edge image point cloud of the edges in the initial bird's-eye view is obtained. The bird's-eye view vehicle transformation matrix is ​​determined based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device. Based on the visible range of the obtained bird's-eye view, the preset scaling ratio, and the bird's-eye view vehicle transformation matrix, the target bird's-eye view transformation matrix is ​​determined.

2. The method according to claim 1, characterized in that, Before performing bird's-eye view calibration processing based on the pixel coordinates of the identifier in the calibration image and the actual coordinates of the identifier in the calibration checkerboard to obtain the initial bird's-eye view transformation matrix, the method further includes: Identify the corner points of the identification code in the calibration image to obtain the pixel coordinates of the corner points; Based on the pixel coordinates, the actual scale between the calibration image and the calibration checkerboard, and the checkerboard size information, the actual coordinates of the corner point of the identification code in the calibration checkerboard are determined.

3. The method according to claim 1, characterized in that, The step of performing bird's-eye view calibration processing based on the pixel coordinates of the identifier in the calibration image and the actual coordinates of the identifier in the calibration checkerboard to obtain an initial bird's-eye view transformation matrix includes: Obtain the preset zoom level and the visible range of the bird's-eye view; The real coordinates are converted into bird's-eye image coordinates based on the scaling ratio and the visible range of the bird's-eye view. The initial bird's-eye view transformation matrix is ​​determined based on the pixel coordinates of the identification code corner point and the bird's-eye view image coordinates.

4. The method according to claim 3, characterized in that, The step of determining the initial bird's-eye view transformation matrix based on the pixel coordinates of the identified corner point and the bird's-eye view image coordinates includes: Obtain the corner points of the multiple identification codes; Multiple target identifier corner points are selected from the multiple identifier corner points according to a preset random sampling consensus algorithm; Based on the pixel coordinates of multiple target identification corner points and the bird's-eye view coordinates, multiple initial transformation matrices and the reprojection error of each initial transformation matrix are determined; The initial bird's-eye view transformation matrix is ​​determined from a plurality of initial transformation matrices based on the reprojection error.

5. The method according to claim 1, characterized in that, The step of obtaining an initial bird's-eye view of the calibrated checkerboard grid based on the initial bird's-eye view transformation matrix, and obtaining the edge image point cloud of the edges in the initial bird's-eye view, includes: The calibration image is transformed according to the initial bird's-eye view transformation matrix to obtain the initial bird's-eye view; Identify a specified corner point in the initial bird's-eye view; The edge corner points of the calibration chessboard in the initial bird's-eye view are determined based on the chessboard size information and the specified corner points; The edge image point cloud is obtained by selecting line segment points based on the baffle size information of the preset baffle, the edge corner points, and the preset distance.

6. The method according to claim 5, characterized in that, The step of selecting line segment points based on the baffle size information of the preset baffle, the edge corner points, and a preset distance to obtain the edge image point cloud includes: The coordinates of the edge corner points are transformed according to the initial bird's-eye view transformation matrix to obtain the true edge corner points; The edge points of the baffle are determined based on a preset linear interpolation algorithm, the actual edge corner points, and the baffle size information; Based on the preset distance, the line segment point is selected from the line segment between the edge point of the baffle and the actual edge corner point to obtain the edge image point cloud.

7. The method according to claim 1, characterized in that, The step of determining the bird's-eye view vehicle transformation matrix based on the edge image point cloud, the baffle laser point cloud, and the laser extrinsic parameters pre-calibrated by the laser device includes: The edge image point cloud and the baffle laser point cloud are registered according to a preset distance iteration algorithm to obtain the point cloud transformation matrix between the baffle laser point cloud and the edge image point cloud. The bird's-eye view vehicle transformation matrix is ​​determined based on the extrinsic parameters of the laser device and the point cloud transformation matrix.

8. The method according to claim 1, characterized in that, The step of determining the target bird's-eye view transformation matrix based on the obtained bird's-eye view visible range, preset zoom ratio, and the bird's-eye view vehicle transformation matrix includes: Based on the bird's-eye view vehicle body transformation matrix, the identification code corner point is transformed to the vehicle body coordinate system where the mobile device is located, and the current coordinates of the identification code corner point in the vehicle body coordinate system are obtained; The current coordinates are converted according to the visible range of the bird's-eye view and the preset zoom ratio to obtain the bird's-eye view coordinates; The target bird's-eye view transformation matrix is ​​obtained by calibration processing based on the bird's-eye view coordinates and the pixel coordinates of the identification code corner points.

9. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images