Automatic driving camera calibration method and system based on virtual calibration workshop

Abstract

The invention discloses an automatic driving camera calibration method and system based on a virtual calibration workshop, and belongs to the technical field of auxiliary driving testing. Comprising the following steps: acquiring parameter information of an automatic driving vehicle, an automatic driving camera and a calibration board, and constructing a virtual vehicle, a virtual calibration board and a virtual camera; virtual calibration plate images at different positions are collected through a virtual camera, marked corner extraction is carried out on the virtual calibration plate images, and multiple sets of calibration parameter initial values of the automatic driving camera are obtained according to image coordinates and position coordinates of marked corners; and based on the initial values of the plurality of groups of calibration parameters, taking the minimum re-projection error as a target, carrying out iterative optimization through an improved bald eagle search algorithm, and determining an optimal calibration parameter. The whole calibration process does not need to depend on a real camera and a physical environment, so that the dependence of a calibration engineer on a hardware environment is reduced; the problem that an existing vehicle needs to depend on a real environment to carry out automatic driving camera calibration work is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of assisted driving test technology, and in particular to an automatic driving camera calibration method and system based on a virtual calibration workshop. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Autonomous driving requires various sensors to perceive the surrounding environment. There is a corresponding conversion relationship between the coordinates on the sensor data (images, point clouds, etc.) and the coordinates of objects in the real world. This conversion relationship can be calculated through formulas obtained through modeling. Some of these formulas contain external parameters of the sensor, and some also contain internal parameters of the sensor; the external parameters are mainly related to the installation orientation of the sensor, and the internal parameters are mainly related to internal factors such as focal length and laser transmitter coordinates. The calibration of the sensor is to derive the internal and external parameters of the sensor through experiments, so as to achieve the unification of the coordinates of each sensor.

[0004] The camera's internal parameters mainly include focal length, mirror distortion level, scaling factor, principal point, etc. The most widely used internal parameter model is Zhang Zhengyou's pinhole model, while cameras with a wide field of view and large distortion will choose fisheye models or panoramic models. The most widely used internal parameter calibration of the camera in the industry is the "Zhang Zhengyou calibration method", which calculates the camera's internal parameters by collecting coordinate data of chessboard calibration plate images at different angles.

[0005] Existing vehicles need to rely on real environments to perform self-driving camera calibration. This will, on the one hand, lead to a longer calibration cycle and higher costs; on the other hand, it increases the time cost of the calibration process. Summary of the invention

[0006] In order to address the deficiencies in the prior art, the present invention provides an autonomous driving camera calibration method, system, electronic device, computer-readable storage medium and computer program product based on a virtual calibration workshop. The autonomous driving camera calibration is performed based on the virtual calibration workshop, which reduces the calibration engineer's dependence on the hardware environment and improves the calibration efficiency.

[0007] In a first aspect, the present invention provides an autonomous driving camera calibration method based on a virtual calibration workshop;

[0008] A method for calibrating an autonomous driving camera based on a virtual calibration workshop, comprising:

[0009] Obtain parameter information of the autonomous driving vehicle, autonomous driving camera and calibration board and construct a virtual vehicle, virtual calibration board and virtual camera;

[0010] The virtual calibration plate images at different positions are collected by the virtual camera, the marked corner points of the virtual calibration plate images are extracted, and multiple sets of initial values ​​of calibration parameters of the autonomous driving camera are obtained according to the image coordinates and position coordinates of the marked corner points;

[0011] Based on multiple groups of initial values ​​of calibration parameters and aiming at minimizing the reprojection error, the optimal calibration parameters are determined by iterative optimization using an improved vulture search algorithm.

[0012] In some implementations, the extraction of marked corner points from the virtual calibration plate image is specifically performed as follows:

[0013] Processing the virtual calibration plate image through a trained image segmentation network to obtain a marked segmentation image;

[0014] Detect and locate the marker corner points in the marker segmentation image and extract the marker corner point position information.

[0015] In some embodiments, the image segmentation network is U 2 Net network.

[0016] In some embodiments, parameter information of the autonomous driving vehicle, the autonomous driving camera, and the calibration plate is processed through a virtual simulation platform to generate a virtual vehicle, a virtual camera, and a virtual calibration plate.

[0017] In some embodiments, the determining the optimal calibration parameters by iteratively optimizing based on multiple sets of initial values ​​of calibration parameters with the goal of minimizing reprojection errors through an improved vulture search algorithm comprises:

[0018] Construct the reprojection error fitness function, determine the solution space based on multiple sets of calibration parameter initial values, and initialize the population size and number of iterations;

[0019] Calculate the current fitness and determine whether the termination condition is met. If so, output the result. If not, update the vulture position and fitness until the termination condition is met.

[0020] Among them, during the process of vulture position updating, the position update weight parameter is adaptively adjusted according to the number of iterations.

[0021] In some implementations, the location update weight parameter is expressed as:

[0022]

[0023] In the formula, t represents the current iteration number, t max Indicates the maximum number of iterations.

[0024] In a second aspect, the present invention provides an autonomous driving camera calibration system based on a virtual calibration workshop;

[0025] An autonomous driving camera calibration system based on a virtual calibration workshop, comprising:

[0026] A virtual calibration workshop construction module is configured to: obtain parameter information of the autonomous driving vehicle, the autonomous driving camera, and the calibration board and construct a virtual vehicle, a virtual calibration board, and a virtual camera;

[0027] The preliminary calibration module is configured to: collect virtual calibration plate images at different positions through a virtual camera, extract marked corner points of the virtual calibration plate images, and obtain multiple sets of initial values ​​of calibration parameters of the autonomous driving camera according to the image coordinates and position coordinates of the marked corner points;

[0028] The optimization module is configured to: determine the optimal calibration parameters by iterative optimization through the improved vulture search algorithm based on multiple groups of initial values ​​of calibration parameters and taking the minimum reprojection error as the goal.

[0029] In a third aspect, the present invention provides an electronic device;

[0030] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned automatic driving camera calibration method based on a virtual calibration workshop.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0032] A computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned automatic driving camera calibration method based on a virtual calibration workshop.

[0033] In a fifth aspect, the present invention provides a computer program product;

[0034] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned automatic driving camera calibration method based on a virtual calibration workshop.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The technical solution provided by the present invention creates a three-dimensional virtual calibration workshop through the technology of the virtual simulation platform. A virtual vehicle equipped with a virtual camera is placed in the center of the workshop, and a virtual ChArUco calibration plate is inserted around the virtual vehicle. At the same time, the spatial layout of the real workshop is simulated to obtain more realistic virtual calibration data.

[0037] 2. The technical solution provided by the present invention does not need to rely on the real camera and physical environment during the entire calibration process, which reduces the calibration engineer's dependence on the hardware environment and greatly reduces the time, money and manpower costs in the calibration process.

[0038] 3. The technical solution provided by the present invention, during the calibration calculation process, through U 2 Net network performs image segmentation on the input image and separates the background area, which effectively improves the detection accuracy and speed of marked corner point target detection; in the iterative optimization process of calibration parameters, the position update weight parameters are adaptively adjusted to improve the optimization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] Figure 1 A schematic diagram of the architecture of a virtual calibration workshop provided in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the flow of an autonomous driving camera calibration method based on a virtual calibration workshop provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0045] Embodiment 1

[0046] The existing autonomous driving camera calibration relies on a real calibration workshop to perform calibration work, which affects the calibration efficiency; therefore, the present invention provides an autonomous driving camera calibration method based on a virtual calibration workshop, which performs real simulation of the autonomous driving camera and calibration board based on a virtual simulation platform, and restores the calibration scene for virtual calibration.

[0047] Next, combine Figure 1-Figure 2 , a method for calibrating an autonomous driving camera based on a virtual calibration workshop disclosed in this embodiment is described in detail. The method for calibrating an autonomous driving camera based on a virtual calibration workshop comprises the following steps:

[0048] S1. Obtain parameter information of the autonomous driving vehicle, autonomous driving camera and calibration board and construct a virtual vehicle, virtual calibration board and virtual camera.

[0049] Specifically, a virtual calibration workshop is created through the virtual simulation platform VTD. Specifically, the structural parameter information of the autonomous driving vehicle is input into the virtual simulation platform VTD, and the virtual vehicle can be automatically generated using its built-in functional components; the structural parameter information, optical parameter information and installation position information of the autonomous driving camera on the autonomous driving vehicle are input into the virtual simulation platform VTD, and the virtual camera installed on the virtual vehicle can be automatically generated using its built-in functional components; the size information and marking information of the calibration plate are input into the virtual simulation platform VTD, and the virtual calibration plate can be automatically generated using its built-in functional components; static scenes are edited through the software ScenarioEditor and RoadDesigner.

[0050] The virtual camera simulated by the virtual simulation platform VTD has the same photo-taking function as a real camera.

[0051] Combination Figure 1 In this embodiment, the virtual calibration board is a virtual ChArUco calibration board, in which all light checker fields are uniquely encoded and identifiable, which means that even partially occluded or non-ideal camera images can be used for calibration. For example, a strong ring light may produce uneven illumination (semi-specular reflection area) on the calibration target, which will cause ordinary checkerboard detection to fail. With ChArUco, the remaining (good) saddle point detection can still be used.

[0052] Furthermore, the virtual vehicle is created by VTD, and the position data of the real vehicle intelligent driving domain sensors are transmitted to the virtual camera and calibration calculation module through the sensor. The virtual camera includes a general measurement, calibration driver (XCP driver) and a virtual controller area network controller (CAN, Controller Area Network) device. The virtual CAN device includes a register read and write interface, a register implementation module, a send data processing module, a receive data processing module and a virtual CAN bus interface call module. The calibration host computer includes an XCP driver.

[0053] Among them, the calibration data is transmitted bidirectionally between the calibration host computer and the virtual camera via the virtual CAN bus. The virtual CAN device calls the virtual CAN bus interface through the data processing module to receive the calibration data sent by the calibration host computer; then, the register read and write interface is called by the register implementation module to store or retrieve the relevant parameters of the calibration data into or out of the register and storage space according to the corresponding category; when the calibration data result needs to be returned, the result is sent to the calibration host computer via the virtual CAN bus by calling the virtual CAN bus interface through the sending data processing module.

[0054] S2, using a virtual camera to collect virtual calibration plate images at different positions, and extracting the marked corner points of the virtual calibration plate images. Specifically including:

[0055] S201, collecting virtual calibration plate images at different positions through a virtual camera.

[0056] Specifically, the virtual calibration plate around the virtual vehicle is photographed by a virtual camera to obtain multiple virtual calibration plate images.

[0057] S202: Process the virtual calibration plate image through the trained image segmentation network to obtain a marked segmentation image.

[0058] Furthermore, the image segmentation network is U 2 Net network, including five first residual U-shaped blocks, one second residual U-shaped block and five third residual U-shaped blocks connected in sequence.

[0059] The first residual U-block and the third residual U-block have the same structure. The first residual U-block includes two first convolution units, five second convolution units, an intermediate layer, one third convolution unit, five fourth convolution units and one addition layer connected in sequence; at the same time, the output of the first first convolution unit to the input of the fifth second convolution unit are respectively the input of the third convolution unit to the addition layer, forming a residual connection. The first convolution unit includes Conv2d+BN+Relu, the second convolution unit includes a downsampling layer and Conv2d+BN+Relu, the intermediate layer includes Conv2d+BN+Relu, the third convolution unit includes a Conct layer and Conv2d+BN+Relu, and the fourth convolution unit includes an upsampling layer, a Conct layer and Conv2d+BN+Relu.

[0060] The second residual U-shaped block includes 8 convolution blocks and an addition layer connected in sequence. At the same time, the output of the first convolution block to the output of the fourth convolution block are the outputs of the sixth convolution block to the addition layer, respectively, forming a residual connection; the convolution block includes Conv2d+BN+Relu.

[0061] Based on the above image segmentation network, a large number of virtual calibration plate images are collected and the image annotation tool Labelme is used to annotate the marked corner point areas in the virtual calibration plate images to obtain binary label images, and a training set is constructed. The image segmentation network is trained with the training set until the training is completed.

[0062] Specifically, the original image and its corresponding binary label image are used as input, and prediction is performed through the image segmentation network to create a pixel-level mask for each marked corner point area.

[0063] In summary, in this embodiment, using U 2 Net network realizes the segmentation of background and markers in the virtual calibration plate image and reduces the interference of background noise on the positioning of marker corners.

[0064] S203, extracting the image coordinates of the marked corner points in the marked segmented image.

[0065] Specifically, the marked corner points in the positioning mark segmentation image are monitored through the OpenCV visual library to obtain the corresponding image coordinates.

[0066] S3. According to the coordinates of the marked corner points in the image and the position coordinates in the world coordinate system, multiple sets of initial values ​​of calibration parameters of the autonomous driving camera are obtained through the calibration parameter initial value obtaining method in Zhang Zhengyou's calibration method.

[0067] Here, the relationship between the image coordinates of the marked corner points and their position coordinates in the world coordinate system is expressed as:

[0068]

[0069] Where s represents the scale factor, A represents the camera intrinsic parameter matrix, including affine transformation and perspective projection; (u0, v0) is the image principal point coordinate, α, β represent the fusion of focal length and pixel aspect ratio, γ represents the radial distortion parameter, R represents the rotation matrix, t represents the translation vector, and [R t] represents the camera extrinsic parameter matrix.

[0070] Zhang Zhengyou's calibration method is a commonly used calibration method in this field. This embodiment does not improve it and will not be described in detail here.

[0071] S4. Based on multiple sets of initial values ​​of calibration parameters, with the goal of minimizing the reprojection error, the optimal calibration parameters are determined by iterative optimization through the improved vulture search algorithm. Specifically, it includes:

[0072] S401, constructing a reprojection error fitness function, determining a solution space based on multiple sets of calibration parameter initial values, initializing the population size and the number of iterations; wherein the reprojection error fitness function is expressed as:

[0073]

[0074] Where n is the number of marked corner points, P i is the image coordinate of the marked corner point, P j Represents the projected point coordinates of the marked corner points.

[0075] S402, calculate the current fitness, determine whether the termination condition is met, if so, output the result, if not, update the vulture position (i.e., calibration parameters) and update the fitness until the termination condition is met, and output the optimal calibration parameters; the position update formula is expressed as:

[0076] P i，new =P best +α*r(P mean -P i );

[0077] Where P best represents the best position of the vulture at present, P mean represents the average position of the vulture after the previous search, P i represents the location of the i-th vulture, α represents the position update weight parameter, and its value is between 1.5 and 2; r is a random number between 0 and 1.

[0078]

[0079] In the formula, t represents the current iteration number, t max Indicates the maximum number of iterations.

[0080] S5. Send the optimal calibration parameters to the calibration host computer through the virtual CAN bus.

[0081] Embodiment 2

[0082] This embodiment discloses an automatic driving camera calibration system based on a virtual calibration workshop, including:

[0083] A virtual calibration workshop construction module is configured to: obtain parameter information of the autonomous driving vehicle, the autonomous driving camera, and the calibration board and construct a virtual vehicle, a virtual calibration board, and a virtual camera;

[0084] The preliminary calibration module is configured to: collect virtual calibration plate images at different positions through a virtual camera, extract marked corner points of the virtual calibration plate images, and obtain multiple sets of initial values ​​of calibration parameters of the autonomous driving camera according to the image coordinates and position coordinates of the marked corner points;

[0085] The optimization module is configured to: determine the optimal calibration parameters by iterative optimization through the improved vulture search algorithm based on multiple groups of initial values ​​of calibration parameters and taking the minimum reprojection error as the goal.

[0086] It should be noted that the virtual calibration workshop construction module, preliminary calibration module, and optimization module described above correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0087] Embodiment 3

[0088] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned automatic driving camera calibration method based on a virtual calibration workshop are completed.

[0089] Embodiment 4

[0090] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned automatic driving camera calibration method based on a virtual calibration workshop are completed.

[0091] Embodiment 5

[0092] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned automatic driving camera calibration method based on a virtual calibration workshop.

[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The automatic driving camera calibration method based on the virtual calibration workshop is characterized by: include: Obtain parameter information of the autonomous driving vehicle, autonomous driving camera and calibration board and construct a virtual vehicle, virtual calibration board and virtual camera; The virtual calibration plate images at different positions are collected by the virtual camera, the marked corner points of the virtual calibration plate images are extracted, and multiple sets of initial values ​​of calibration parameters of the autonomous driving camera are obtained according to the image coordinates and position coordinates of the marked corner points; Based on multiple groups of initial values ​​of calibration parameters and aiming at minimizing the reprojection error, the optimal calibration parameters are determined by iterative optimization using an improved vulture search algorithm.

2. The automatic driving camera calibration method based on a virtual calibration workshop according to claim 1, characterized in that: The extraction of marked corner points from the virtual calibration plate image is specifically as follows: Processing the virtual calibration plate image through a trained image segmentation network to obtain a marked segmentation image; Detect and locate the marker corner points in the marker segmentation image and extract the marker corner point position information.

3. The automatic driving camera calibration method based on a virtual calibration workshop according to claim 2, characterized in that: The image segmentation network is U 2 Net network.

4. The automatic driving camera calibration method based on a virtual calibration workshop according to claim 1, characterized in that: The parameter information of the autonomous driving vehicle, autonomous driving camera and calibration board is processed through the virtual simulation platform to generate a virtual vehicle, a virtual camera and a virtual calibration board.

5. The automatic driving camera calibration method based on a virtual calibration workshop according to claim 1, characterized in that: The method of determining the optimal calibration parameters based on multiple groups of initial values ​​of calibration parameters and minimizing the reprojection error is performed by iterative optimization through an improved vulture search algorithm, and includes: Construct the reprojection error fitness function, determine the solution space based on multiple sets of calibration parameter initial values, and initialize the population size and number of iterations; Calculate the current fitness and determine whether the termination condition is met. If so, output the result. If not, update the vulture position and fitness until the termination condition is met. Among them, during the process of vulture position updating, the position update weight parameter is adaptively adjusted according to the number of iterations.

6. The automatic driving camera calibration method based on a virtual calibration workshop according to claim 5, characterized in that: The position update weight parameter is expressed as: In the formula, t represents the current iteration number, t max Indicates the maximum number of iterations.

7. The automatic driving camera calibration system based on the virtual calibration workshop is characterized by: include: A virtual calibration workshop construction module is configured to: obtain parameter information of the autonomous driving vehicle, the autonomous driving camera, and the calibration board and construct a virtual vehicle, a virtual calibration board, and a virtual camera; The preliminary calibration module is configured to: collect virtual calibration plate images at different positions through a virtual camera, extract marked corner points of the virtual calibration plate images, and obtain multiple sets of initial values ​​of calibration parameters of the autonomous driving camera according to the image coordinates and position coordinates of the marked corner points; The optimization module is configured to: determine the optimal calibration parameters by iterative optimization through the improved vulture search algorithm based on multiple groups of initial values ​​of calibration parameters and taking the minimum reprojection error as the goal.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the automatic driving camera calibration method based on a virtual calibration workshop as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the automatic driving camera calibration method based on a virtual calibration workshop as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the automatic driving camera calibration method based on a virtual calibration workshop as described in any one of claims 1 to 6 are implemented.

Images