Camera dynamic calibration method, vehicle, electronic equipment and program product

Abstract

The invention provides a camera dynamic calibration method, a vehicle, electronic equipment and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: when a camera dynamic calibration starting condition is satisfied, acquiring a reference acquisition image of a reference camera of a target vehicle and a target acquisition image of a target camera; determining a lane line point set of each lane line according to the reference acquisition image, and determining an external parameter matrix of the reference camera based on the lane line point sets; the lane line point set comprises lane line pixel position coordinates of corresponding lane lines; performing feature matching based on common-view areas of the reference acquisition image and the target acquisition image acquired at the same moment to obtain a target matching feature point pair set; and determining an external parameter matrix of the target camera according to the target matching feature point pair set and the external parameter matrix of the reference camera. According to the invention, the driving camera calibration of the intelligent driving vehicle can give consideration to both efficiency and precision.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a camera dynamic calibration method, a vehicle, an electronic device, and a program product. Background Art

[0002] Camera after-sales calibration is an important after-sales process for machine vision products such as driverless vehicles and mobile robots. In the after-sales dynamic calibration schemes for intelligent driving vehicle driving cameras, some have high requirements for road conditions and perception algorithm performance in order to ensure calibration accuracy. Calibration often fails because road conditions cannot continuously meet the requirements, or because the required perception algorithm consumes a lot of resources and is difficult to deploy for mass production; some get rid of the constraints of road ground elements and perception algorithms in pursuit of efficiency, resulting in unstable calibration accuracy and affecting the safety of intelligent driving functions. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a camera dynamic calibration method, a vehicle, an electronic device, and a program product, so that the calibration of the driving camera of an intelligent driving vehicle can take into account both efficiency and accuracy.

[0004] In a first aspect, the present disclosure provides a camera dynamic calibration method, comprising:

[0005] When the camera dynamic calibration start condition is met, a reference acquisition image of a reference camera of a target vehicle and a target acquisition image of a target camera are acquired;

[0006] Determine a lane line point set for each lane line according to the reference captured image, and determine an extrinsic parameter matrix of the reference camera based on the lane line point set; the lane line point set includes lane line pixel position coordinates of the corresponding lane line;

[0007] Perform feature matching based on the common viewing area of ​​the reference captured image and the target captured image captured at the same time to obtain a target matching feature point pair set;

[0008] The extrinsic parameter matrix of the target camera is determined according to the target matching feature point pair set and the extrinsic parameter matrix of the reference camera.

[0009] In a second aspect, the present disclosure provides an electronic device, including:

[0010] at least one processor; and

[0011] a memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores at least one computer program executable by the at least one processor, and the at least one computer program is executed by the at least one processor so that the at least one processor can execute the camera dynamic calibration method as described in the first aspect.

[0013] In a third aspect, the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed in a processor, the camera dynamic calibration method described in the first aspect is implemented.

[0014] Optionally, the computer program may be stored in a readable storage medium of a computer device or in the cloud; and the processor of the computer device reads the computer program from the readable storage medium or in the cloud.

[0015] In a fourth aspect, the present disclosure provides a vehicle, wherein the vehicle is configured to execute the camera dynamic calibration method described in the first aspect.

[0016] In the embodiment provided by the present disclosure, when the conditions for starting the dynamic calibration of the camera are met, the reference acquisition image of the reference camera of the target vehicle and the target acquisition image of the target camera are obtained, the lane line point set of each lane line is determined according to the reference acquisition image, and the external parameter matrix of the reference camera is determined based on the lane line point set. The external parameter matrix of the reference camera is determined by a set of lane line detection methods based on the image, without relying on a high computing power platform, making full use of road elements, and realizing efficient calibration of the reference camera. In addition, feature matching is performed based on the common viewing area of ​​the reference acquisition image and the target acquisition image acquired at the same time to obtain a set of target matching feature point pairs; the external parameter matrix of the target camera is determined according to the set of target matching feature point pairs and the external parameter matrix of the reference camera. By matching the feature points, the external parameter matrix of the target camera can be efficiently determined by using the external parameter matrix of the calibrated reference camera, which also does not need to rely on a high computing power platform, does not require a complicated processing process, and can efficiently complete the calibration of the target camera associated with the reference camera. In summary, the present disclosure can complete the calibration of the periscopic camera on intelligent driving vehicles with medium and low computing power platforms, taking into account calibration accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0018] Figure 1 The figure is a schematic diagram of the process of the camera dynamic calibration method in the embodiment of the present disclosure;

[0019] Figure 2 The figure shows a schematic diagram of the angle formed by the lane line and the intersection in the embodiment of the present disclosure;

[0020] Figure 3 The figure is a schematic diagram of the flow of the calibration method for the driving camera of an intelligent driving vehicle in an embodiment of the present disclosure;

[0021] Figure 4 The block diagram of the camera dynamic calibration device in the embodiment of the present disclosure is shown;

[0022] Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0024] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0025] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0026] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0028] Exemplary Methods

[0029] The camera dynamic calibration method provided by the embodiment of the present disclosure may be applied to a main controller of a vehicle, or may be applied to other terminals capable of communicating with the vehicle and / or sensors installed on the vehicle.

[0030] The camera dynamic calibration method provided by the embodiment of the present disclosure is as follows: Figure 1 As shown, it mainly includes the following steps:

[0031] Step 101 : when the camera dynamic calibration start condition is met, a reference captured image of a reference camera of a target vehicle and a target captured image of a target camera are acquired.

[0032] In some embodiments, the reference camera includes a front-view camera, and the target camera includes a left front-view camera or a right front-view camera; and / or, the reference camera includes a rear-view camera, and the target camera includes a left rear-view camera or a right rear-view camera.

[0033] In some embodiments, satisfying the camera dynamic calibration start condition includes satisfying the dynamic road calibration condition, and the dynamic road calibration condition includes: satisfying the external road environment condition and the vehicle driving state condition.

[0034] Meeting the external road environment conditions means that the external road environment is flat, has no steep slopes, no road bumps, has multiple clear lane lines, has rich features on both sides of the road, such as static objects such as trees and buildings, and has good lighting conditions during the day.

[0035] Meeting the driving state conditions of the vehicle means that the vehicle is in human driving mode, the speed is between 30km / h and 80km / h, it is in a straight line in the middle of the lane, there is no frequent left and right turns, no frequent sudden acceleration or deceleration, and no vehicle bumps. At the same time, the vehicle's two-dimensional (2D) lane line detection function can continuously, accurately and stably output lane line point sets.

[0036] Step 102, determining a lane line point set of each lane line according to the reference captured image, and determining an extrinsic parameter matrix of the reference camera based on the lane line point set; the lane line point set includes lane line pixel position coordinates of the corresponding lane line.

[0037] In some embodiments, determining a lane line point set of each lane line based on the reference captured image includes: detecting the lane line point set in the reference captured image through a 2D lane line detection model, wherein the lane line point set includes lane line pixel position coordinates.

[0038] The 2D lane line detection model may be obtained by pre-training a learning model, and there is no restriction on the specific type of the learning model.

[0039] In some embodiments, a reference captured image can detect multiple lane line point sets, one lane line point set corresponds to one lane line, including lane line pixel position coordinates belonging to the lane line. The lane line point set of each lane line can be independently indexed.

[0040] Here, each pixel point in the lane line point set is given in the form of 2D lane line pixel position coordinates, rather than 3D perception of the lane line to detect the real physical space position of the lane line, avoiding the high computational complexity brought by 3D perception and improving detection efficiency.

[0041] In some embodiments, determining the extrinsic parameter matrix of the reference camera based on the lane line point set includes: fitting a mathematical expression of each lane line in the image coordinate system according to the lane line point set in the reference captured image; determining the intersection coordinates of every two lane lines in the image coordinate system based on the mathematical expression of each lane line to obtain a set of intersection coordinates; determining an initial extrinsic parameter matrix of the reference camera based on the intersection coordinate set; the initial extrinsic parameter matrix includes a pitch angle, a yaw angle, and a default roll angle; optimizing the initial extrinsic parameter matrix based on the mathematical expression of each lane line to obtain the optimized extrinsic parameter matrix.

[0042] The mathematical expression of the lane line is obtained by fitting the lane line point set, and any fitting algorithm may be used. The specific fitting algorithm used is not limited here.

[0043] Exemplarily, after fitting the mathematical expression of each lane line in the image coordinate system according to the lane line point set in the reference acquisition image, the method further includes: removing lane lines that do not meet the conditions according to the mathematical expression of each lane line, the conditions including that the lane line is straight, stable driving, etc.

[0044] Lane lines that do not meet the conditions are removed, including but not limited to: 1. Lane lines whose parameters describing curvature in mathematical expressions are greater than a certain threshold are removed; 2. Lane lines whose points are concentrated with lane line bifurcations, i.e., one lane line changes into two lane lines, or two lane lines change into one lane line, are removed; 3. Lane lines corresponding to when the vehicle is in an uphill, downhill or continuously bumpy state obtained through the vehicle's inertial navigation system (INS), inertial measurement unit (IMU) or combined inertial navigation information are removed; 4. Lane lines whose length is less than the length threshold are removed.

[0045] Exemplarily, based on the mathematical expression of each lane line, the coordinates of the intersection of every two lane lines in the image coordinate system are determined, including but not limited to the following methods:

[0046] Method 1: Solve the linear regression equation of the lane line point set, and calculate the intersection coordinates through the linear regression equation of the two lane lines;

[0047] Method 2: Obtain two points in the preset area of ​​the lane point set, use them as known conditions to solve the straight line equation passing through the two points, and use the straight line equation representing the lane as the representative straight line equation, and calculate the intersection coordinates through the respective representative straight line equations of the two lane lines;

[0048] Method 3: Solve all the straight line equations from the starting point of the lane line point set to any other point in the lane line point set, count the slope and bias parameters of the straight line equation, remove the maximum and minimum values ​​of all slopes and all bias parameters, and then calculate the average, substitute the slope average and bias parameter average into the straight line equation to obtain the characterization straight line equation for the lane line, and finally calculate the intersection coordinates through the characterization straight line equations of the two lane lines. Among them, the point closest to the bottom of the image in each lane line point set is the starting point.

[0049] Exemplarily, determining the initial extrinsic parameter matrix of the reference camera based on the intersection point coordinate set includes: determining optimal intersection point coordinates based on the intersection point coordinate set, and determining the initial extrinsic parameter matrix of the reference camera according to the optimal intersection point coordinates and the intrinsic parameter matrix of the reference camera.

[0050] Exemplarily, before determining the initial external parameter matrix of the reference camera based on the optimal intersection point coordinates and the internal parameter matrix of the reference camera, it also includes: after proposing the external points in the intersection point coordinate set based on known prior conditions, the known prior conditions include the tooling posture and tooling accuracy of the reference camera.

[0051] In some embodiments, the method further includes: after the external parameter matrices corresponding to each of the N frames of the benchmark acquisition images are obtained, voting on the N external parameter matrices, and determining the final external parameter matrix of the benchmark camera according to the voting results; N is an integer greater than 1.

[0052] In some embodiments, the optimizing the initial extrinsic parameter matrix based on the mathematical expression of each lane line to obtain the extrinsic parameter matrix of the reference camera includes: determining the angle of the lane line relative to the horizontal coordinate axis of the image coordinate system based on the mathematical expression of each lane line; optimizing the roll angle in the initial extrinsic parameter matrix according to the angle corresponding to each lane line to obtain the extrinsic parameter matrix of the reference camera.

[0053] like Figure 2As shown, the angle between lane line LineLeft_i and the horizontal coordinate axis of the image coordinate system (i.e., the horizontal dotted line) is θ1, the angle between lane line LineRight_i and the horizontal coordinate axis of the image coordinate system (i.e., the horizontal dotted line) is θ2, the intersection point of LineLeft_i and LineRight_i is VP, and the angle formed by LineLeft_i and LineRight_i at the intersection point is θ3.

[0054] Under ideal conditions, θ1 and θ2 are equal. Here, the roll angle of the reference camera is optimized according to the difference between θ1 and θ2. The larger the difference between θ1 and θ2, the larger the optimized roll angle.

[0055] Exemplarily, based on the statistically determined angles belonging to the same lane line, the optimal angle of the lane line is determined, and based on the optimal angle of each lane line, the roll angle in the initial extrinsic parameter matrix is ​​optimized.

[0056] Step 103 : performing feature matching based on the common viewing area of ​​the reference captured image and the target captured image captured at the same time to obtain a target matching feature point pair set.

[0057] In some embodiments, the performing feature matching on the common viewing area of ​​the reference acquisition image and the target acquisition image acquired at the same time to obtain a target matching feature point pair set includes: cropping the reference acquisition image and the target acquisition image to obtain a first intermediate image group; the first intermediate image group includes a reference intermediate image cropped from the reference acquisition image and a target intermediate image cropped from the target acquisition image, and the reference intermediate image and the target intermediate image both include the common viewing area of ​​the reference acquisition image and the target acquisition image; for the first intermediate image group, detecting feature points of the reference intermediate image and the target intermediate image, and performing feature matching based on the feature points of the reference intermediate image and the target intermediate image to obtain a target matching feature point pair set.

[0058] In order to ensure processing consistency, computational efficiency, and detection stability, the image sizes in the first intermediate image group are kept consistent and the image range includes the common viewing area of ​​the cameras associated with it, that is, the common viewing area of ​​the two associated cameras is a subset of the cropped image range.

[0059] In some embodiments, the feature matching is performed based on the feature points of the reference intermediate image and the target intermediate image to obtain a set of target matching feature point pairs, including: determining the degree of matching between the feature point pairs of the reference intermediate image and the target intermediate image, obtaining feature point pairs with a matching degree higher than a threshold, and obtaining a first matching feature point group; determining the norm of the feature point pairs in the first matching feature point group, deleting the feature point pairs whose norm exceeds the norm threshold from the first matching feature point group, and obtaining a second matching feature point group; and using the feature point pairs in the second matching feature point group as target matching feature point pairs to obtain a set of target matching feature point pairs.

[0060] By acquiring feature point pairs with matching degrees higher than a threshold, a first matching feature point group is formed, so that the pairs in the first matching feature point group are the remaining feature point pairs after eliminating the feature point pairs that are misdetected or mismatched, so as to ensure the accuracy of calculation.

[0061] Exemplarily, the norm is an L2 norm in a pixel coordinate system, and the norm threshold is pre-configured according to the field of view range of the images in the first intermediate image group and debugging statistical results.

[0062] Step 104 : determining the extrinsic parameter matrix of the target camera according to the target matching feature point pair set and the extrinsic parameter matrix of the reference camera.

[0063] In some embodiments, determining the extrinsic parameter matrix of the target camera based on the target matching feature point pair set and the extrinsic parameter matrix of the reference camera includes: when the number of target matching feature pairs in the target matching feature point pair set is not less than a preset number, determining a basic matrix based on the target matching feature point pair set; determining feature point pairs in the target matching feature point pair set that do not satisfy epipolar geometric projection based on the basic matrix, and deleting them from the target matching feature point pair set to obtain a third matching feature group; determining a relative extrinsic parameter matrix of the target camera relative to the origin pose of the reference camera based on the third matching feature point group; determining the extrinsic parameter matrix of the target camera based on the extrinsic parameter matrix of the reference camera and the relative extrinsic parameter matrix of the target camera.

[0064] The preset number is pre-configured. Exemplarily, the preset number is 7 or 8, that is, the number of target matching feature point pairs in the target matching feature point pair set is not less than 7 or 8 feature point pairs.

[0065] By filtering out the feature point pairs that do not satisfy the epipolar geometry projection, and again proposing the feature pairs that do not satisfy the projection relationship, the feature point pairs in the third matching feature point group can satisfy the projection relationship, further ensuring the accuracy of the extrinsic parameter calibration.

[0066] Exemplarily, determining the relative extrinsic parameter matrix of the target camera relative to the origin position of the reference camera based on the third matching feature point group includes: using a nonlinear optimization algorithm to solve the relative extrinsic parameter matrix of the target camera based on the origin position of the reference camera based on the third matching feature point group.

[0067] For example, the nonlinear optimization algorithm is solved using the ceres solver library. Specifically, one of the optimization models in the ceres solver library is the bundle adjustment method (BA), which aims to minimize the reprojection error. Given the installation poses of the target camera and the reference camera and the coordinates of each feature point in the third matching feature point group, these feature points are projected from the reference camera image plane to the target camera image plane through the projection equation, and the reprojection error between these predicted points and the actual observation points can be calculated. An optimization algorithm (such as the Gauss-Newton method, Levenberg-Marquardt, and the trust region method) is used to iteratively adjust the extrinsic parameter matrix of the target camera to reduce the reprojection error. In each iteration, the gradient is calculated based on the current error, and the extrinsic parameter matrix of the target camera is updated in the opposite direction of the gradient. This process is repeated until the reprojection error reaches an acceptable level or the preset number of iterations is reached. Finally, a set of optimized camera extrinsic parameter matrices are obtained by iterative calculation. This set of parameters can more accurately describe the extrinsic parameter matrix of the target camera relative to the origin pose of the reference camera.

[0068] Exemplarily, the extrinsic parameter matrix of the target camera is determined according to the extrinsic parameter matrix of the reference camera and the relative extrinsic parameter matrix of the target camera, including: taking the extrinsic parameter matrix of the reference camera as a reference, converting the relative extrinsic parameter matrix of the target camera into the vehicle coordinate system to keep it consistent with the reference camera, and obtaining the extrinsic parameter matrix corresponding to the image captured by the current frame of the target camera.

[0069] For example, when the reference camera is a front-view camera, assuming Represents the transformation matrix from the forward-looking camera coordinate system to the vehicle RFU coordinate system, which is a point P in the forward-looking camera coordinate system. front_cam The coordinates of this point in the vehicle RFU coordinate system can be obtained Assume that the target camera is any circumferential camera (left front-view camera or right front-view camera) that has a common viewing area with the front-view camera. Represents the transformation matrix from the panoramic camera coordinate system to the front camera coordinate system, which is a point P in the panoramic camera coordinate system side_cam The coordinates of the point in the forward-looking camera coordinate system can be obtained So, Represents the transformation matrix from the panoramic camera coordinate system to the ego vehicle RFU coordinate system, From a point P in the panoramic camera coordinate system side_camThe coordinates of this point in the vehicle RFU coordinate system can be obtained When the reference camera is a rear-view camera, the transformation matrix between the rear-view camera, the panoramic camera, and the ego-vehicle RFU coordinate system is similar.

[0070] In some embodiments, the method further includes: when obtaining M extrinsic parameter matrices of the target camera through the reference acquisition image and the target acquisition image acquired at the same time through M consecutive frames, voting on the M extrinsic parameter matrices to determine the final extrinsic parameter matrix of the target camera; M is an integer greater than 1.

[0071] Exemplarily, before voting, the external points in the obtained external parameter matrix are removed to delete abnormal and out-of-limit data. The abnormal and out-of-limit data here refer to the external parameter matrix being identified as an external point when any of the three components of the pitch angle, yaw angle and roll angle of the external parameter matrix exceeds a preset range.

[0072] In some embodiments, after obtaining the extrinsic parameter matrix of the target camera, the extrinsic parameter matrix of the target camera is compared with the tooling posture. If the difference between the two is within the allowable range, an epipolar geometry check is performed based on the extrinsic parameter matrix. If the check passes, the extrinsic parameter matrix is ​​saved and a log is published. If the result of the comparison with the tooling posture is not within the allowable range, the extrinsic parameter matrix and the corresponding error code are saved in the calibration process record. If the epipolar geometry check fails, the extrinsic parameter matrix and the corresponding error code are also saved in the calibration process record.

[0073] In an exemplary embodiment, Figure 3 As shown, the calibration method process for the driving camera of an intelligent driving vehicle mainly includes:

[0074] Step 301, obtaining a lane line point set of each frame image in an image sequence captured by a front-view camera and a rear-view camera;

[0075] Step 302, calculating the extrinsic parameter matrix of the front view camera according to the lane line point set of the front view camera, and calculating the extrinsic parameter matrix of the rear view camera according to the lane line point set of the rear view camera;

[0076] Step 303: for the left front-view camera and the right front-view camera that have a common viewing area with the front-view camera, respectively obtain a first common viewing area image of the left front-view camera and the front-view camera, and a second common viewing area image of the right front-view camera and the front-view camera; and for the left rear-view camera and the right rear-view camera that have a common viewing area with the rear-view camera, respectively obtain a third common viewing area image of the left rear-view camera and the rear-view camera, and a fourth common viewing area image of the right rear-view camera and the rear-view camera;

[0077] Step 304, obtaining, by cropping, a first intermediate image group corresponding to the first common viewing area image, a first intermediate image group corresponding to the second common viewing area image, a first intermediate image group corresponding to the third common viewing area image, and a first intermediate image group corresponding to the fourth common viewing area image;

[0078] Step 305, solving the extrinsic parameter matrix of the left front view camera based on the first intermediate image group corresponding to the first common view area image, solving the extrinsic parameter matrix of the right front view camera based on the first intermediate image group corresponding to the second common view area image, solving the extrinsic parameter matrix of the left rear view camera based on the first intermediate image group corresponding to the third common view area image, and solving the extrinsic parameter matrix of the right rear view camera based on the first intermediate image group corresponding to the fourth common view area image;

[0079] Step 306 , processing the calibration results of each driving camera to complete the calibration process.

[0080] In the embodiment provided by the present disclosure, when the conditions for starting the dynamic calibration of the camera are met, the reference acquisition image of the reference camera of the target vehicle and the target acquisition image of the target camera are obtained, the lane line point set of each lane line is determined according to the reference acquisition image, and the external parameter matrix of the reference camera is determined based on the lane line point set. The external parameter matrix of the reference camera is determined by a set of lane line detection methods based on the image, without relying on a high computing power platform, making full use of road elements, and realizing efficient calibration of the reference camera. In addition, feature matching is performed based on the common viewing area of ​​the reference acquisition image and the target acquisition image acquired at the same time to obtain a set of target matching feature point pairs; the external parameter matrix of the target camera is determined according to the set of target matching feature point pairs and the external parameter matrix of the reference camera. By matching the feature points, the external parameter matrix of the target camera can be efficiently determined by using the external parameter matrix of the calibrated reference camera, which also does not need to rely on a high computing power platform, does not require a complicated processing process, and can efficiently complete the calibration of the target camera associated with the reference camera. In summary, the present disclosure can complete the calibration of the periscopic camera on intelligent driving vehicles with medium and low computing power platforms, taking into account calibration accuracy and efficiency.

[0081] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between the steps is not limited to being implemented according to the step number.

[0082] Exemplary Devices

[0083] Figure 4A block diagram of a camera dynamic calibration device provided by an embodiment of the present disclosure, the vehicle outside temperature determination device mainly comprises: an acquisition module 401, which is used to acquire a reference acquisition image of a reference camera of a target vehicle and a target acquisition image of a target camera when a camera dynamic calibration start condition is met;

[0084] A determination module 402 is used to determine a lane line point set of each lane line according to the reference captured image, and determine an extrinsic parameter matrix of the reference camera based on the lane line point set; the lane line point set includes lane line pixel position coordinates of the corresponding lane line;

[0085] A matching module 403 is used to perform feature matching based on the common viewing area of ​​the reference captured image and the target captured image captured at the same time to obtain a target matching feature point pair set;

[0086] The processing module 404 is used to determine the extrinsic parameter matrix of the target camera according to the target matching feature point pair set and the extrinsic parameter matrix of the reference camera.

[0087] Exemplary Electronic Devices

[0088] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0089] Reference Figure 5 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502, and one or more I / O interfaces 503, connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501, so that the at least one processor 501 can execute the above-mentioned camera dynamic calibration method.

[0090] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0091] Exemplary vehicles, computer program products, and storage media

[0092] The embodiments of the present disclosure also provide a vehicle, which is configured to execute the camera dynamic calibration method as described above.

[0093] The embodiments of the present disclosure further provide a computer program product, including a computer program, which implements the above-mentioned camera dynamic calibration method when executed in a processor.

[0094] The computer program may be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or in the cloud.

[0095] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0096] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0097] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0099] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0100] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0101] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0102] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0103] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0104] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0105] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A camera dynamic calibration method, characterized in that: include: When the camera dynamic calibration start condition is met, a reference acquisition image of a reference camera of a target vehicle and a target acquisition image of a target camera are acquired; Determine a lane line point set for each lane line according to the reference captured image, and determine an extrinsic parameter matrix of the reference camera based on the lane line point set; the lane line point set includes lane line pixel position coordinates of the corresponding lane line; Perform feature matching based on the common viewing area of ​​the reference captured image and the target captured image captured at the same time to obtain a target matching feature point pair set; The extrinsic parameter matrix of the target camera is determined according to the target matching feature point pair set and the extrinsic parameter matrix of the reference camera.

2. The method according to claim 1, characterized in that The determining the extrinsic parameter matrix of the reference camera based on the lane line point set comprises: According to the lane line point set in the reference captured image, a mathematical expression of each lane line in the image coordinate system is obtained by fitting; Based on the mathematical expression of each lane line, determine the intersection coordinates of every two lane lines in the image coordinate system to obtain an intersection coordinate set; Determining an initial extrinsic parameter matrix of the reference camera based on the intersection point coordinate set; the initial extrinsic parameter matrix includes a pitch angle, a yaw angle, and a default roll angle; The initial extrinsic parameter matrix is ​​optimized based on the mathematical expression of each lane line to obtain the optimized extrinsic parameter matrix.

3. The method according to claim 2, characterized in that The method further comprises: After the extrinsic parameter matrices corresponding to the respective N frames of the benchmark acquisition images are obtained, the N extrinsic parameter matrices are voted, and the final extrinsic parameter matrix of the benchmark camera is determined according to the voting results; wherein N is an integer greater than 1.

4. The method according to claim 2, characterized in that: The step of optimizing the initial extrinsic parameter matrix based on the mathematical expression of each lane line to obtain the extrinsic parameter matrix of the reference camera includes: Based on the mathematical expression of each lane line, determining the angle of the lane line relative to the horizontal coordinate axis of the image coordinate system; According to the angle corresponding to each lane line, the roll angle in the initial extrinsic parameter matrix is ​​optimized to obtain the extrinsic parameter matrix of the reference camera.

5. The method according to claim 1, characterized in that The performing feature matching based on the common viewing area of ​​the reference acquired image and the target acquired image acquired at the same time to obtain a target matching feature point pair set includes: The reference acquisition image and the target acquisition image are cropped to obtain a first intermediate image group; the first intermediate image group includes a reference intermediate image cropped from the reference acquisition image and a target intermediate image cropped from the target acquisition image, and both the reference intermediate image and the target intermediate image include a common viewing area of ​​the reference acquisition image and the target acquisition image; For the first intermediate image group, feature points of the reference intermediate image and the target intermediate image are detected, and feature matching is performed based on the feature points of the reference intermediate image and the target intermediate image to obtain a set of target matching feature point pairs.

6. The method according to claim 5, characterized in that The performing feature matching based on the feature points of the reference intermediate image and the target intermediate image to obtain a target matching feature point pair set includes: Determine the matching degree between the feature point pairs of the reference intermediate image and the target intermediate image, obtain the feature point pairs with matching degrees higher than a threshold, and obtain a first matching feature point group; Determine the norm of the feature point pairs in the first matching feature point group, delete the feature point pairs whose norms exceed the norm threshold from the first matching feature point group, and obtain a second matching feature point group; The feature point pairs in the second matching feature point group are used as target matching feature point pairs to obtain a target matching feature point pair set.

7. The method according to claim 1, characterized in that The step of determining the extrinsic parameter matrix of the target camera according to the target matching feature point pair set and the extrinsic parameter matrix of the reference camera comprises: When the number of target matching feature point pairs in the target matching feature point pair set is not less than a preset number, determining a basic matrix according to the target matching feature point pair set; Determine, according to the basic matrix, feature point pairs in the target matching feature point pair set that do not satisfy epipolar geometric projection, and delete them from the target matching feature point pair set to obtain a third matching feature group; Determine, according to the third matching feature point group, a relative extrinsic parameter matrix of the target camera relative to the origin position of the reference camera; The extrinsic parameter matrix of the target camera is determined according to the extrinsic parameter matrix of the reference camera and the relative extrinsic parameter matrix of the target camera.

8. The method according to claim 7, characterized in that The method further comprises: When M extrinsic parameter matrices of the target camera are obtained by using the reference acquired image and the target acquired image acquired at the same time in M ​​consecutive frames, the M extrinsic parameter matrices are voted to determine a final extrinsic parameter matrix of the target camera; M is an integer greater than 1.

9. The method according to any one of claims 1 to 8, characterized in that: The reference camera includes a front-view camera, and the target camera includes a left front-view camera or a right front-view camera; The reference camera includes a rear-view camera, and the target camera includes a left rear-view camera or a right rear-view camera.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program executable by the at least one processor, and the at least one computer program is executed by the at least one processor so that the at least one processor can execute the camera dynamic calibration method according to any one of claims 1 to 9.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed in a processor, the camera dynamic calibration method according to any one of claims 1 to 9 is implemented.

12. A vehicle, characterized in that: The vehicle is configured to execute the camera dynamic calibration method according to any one of claims 1-9.

Images