Camera extrinsic parameter calibration method, device, equipment and storage medium

Abstract

The application relates to a camera extrinsic parameter calibration method, device, equipment and storage medium. The method comprises the following steps: acquiring multiple images shot by a camera; setting first camera extrinsic parameters of the multiple images respectively, wherein the first camera extrinsic parameters of the multiple images and the true value of the camera extrinsic parameters are in normal distribution; calculating a first loss function of each image according to the first camera extrinsic parameters of the multiple images respectively; combining the first loss function of each image and a preset second camera extrinsic parameter to calculate a second loss function of the multiple images; iteratively solving the second loss function of the multiple images until the second loss function of the multiple images is not greater than a preset threshold, and taking the corresponding preset second camera extrinsic parameter as a target camera extrinsic parameter. The scheme provided by the application can take the target extrinsic parameter as the camera extrinsic parameter of any image shot by the camera during vehicle driving, and the target extrinsic parameter approximates the true value and has high accuracy.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to camera extrinsic parameter calibration methods, apparatus, devices and storage media. Background Technology

[0002] Camera extrinsic parameters determine the camera's position and orientation in a three-dimensional space; the camera's pose is also called its extrinsic parameters. Unlike the unchanging intrinsic parameters, extrinsic parameters change with camera movement. Furthermore, extrinsic parameters determine the perception accuracy of autonomous vehicles and the accuracy of high-precision maps. Therefore, how to better calibrate camera extrinsic parameters is a crucial issue in autonomous driving.

[0003] In related technologies, when calibrating camera extrinsics, the camera extrinsics obtained from the previous image are generally used as the initial values ​​for the extrinsics of the current image to calculate the corresponding camera extrinsics for the current image.

[0004] However, in autonomous driving, the camera moves due to the vehicle's vibrations, and the extrinsic parameters of the images captured by the camera are different at different times. The camera extrinsic parameters solved by the above scheme are only applicable to the current frame image and do not approximate the true value, and are not applicable to every image captured by the camera. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides a camera extrinsic parameter calibration method, apparatus, device, and storage medium, which can use the target extrinsic parameter as the camera extrinsic parameter of any image captured by the camera while the vehicle is in motion, and the target extrinsic parameter approximates the true value with high accuracy.

[0006] The first aspect of this application provides a method for calibrating camera extrinsic parameters, including:

[0007] Acquire multiple images captured by the camera;

[0008] Set the first camera extrinsic parameters for each of the multiple images, and ensure that the first camera extrinsic parameters for each of the multiple images and the true values ​​of the camera extrinsic parameters are normally distributed;

[0009] Calculate the first loss function for each image based on the first camera extrinsic parameters of each of the multiple images;

[0010] By combining the first loss function of each image with the preset second camera extrinsic parameters, the second loss function of the multiple images is calculated;

[0011] Change the value of the preset second camera extrinsic parameter, iteratively solve the second loss function of the multiple images until the second loss function of the multiple images is not greater than a preset threshold, and use the corresponding preset second camera extrinsic parameter as the target camera extrinsic parameter.

[0012] As an optional embodiment, the step of calculating the second loss function for the multiple images by combining the first loss function of each image and preset second camera extrinsic parameters includes:

[0013] Based on the first loss function of each image, the sum of the first loss functions of each image is calculated, and the first equation is obtained;

[0014] Calculate the absolute value of the difference between the first camera extrinsic parameter and the preset second camera extrinsic parameter of the i-th image, and multiply the sum of the absolute values ​​of the difference between the first camera extrinsic parameter and the preset second camera extrinsic parameter of the i-th image by a preset weighting coefficient to obtain the second equation; where i = 1, 2, 3...n, and n is the total number of the multiple images;

[0015] Summing the first equation and the second equation yields the second loss function for the multiple images.

[0016] As an optional embodiment, the second loss function of the multiple images is solved iteratively using a gradient descent method.

[0017] As an optional embodiment, acquiring multiple images captured by the camera includes:

[0018] The system periodically acquires multiple images captured by the camera while the vehicle is in motion.

[0019] As an optional embodiment, the vehicle's offset data in each cycle during vehicle movement follows a normal distribution with the true value of the vehicle's position when the vehicle is in a preset direction.

[0020] As an optional embodiment, the true values ​​of the camera extrinsic parameters include the camera's yaw angle, pitch angle, roll angle, and altitude in the world coordinate system.

[0021] As an optional embodiment, calculating a first loss function for each of the plurality of images based on their respective first camera extrinsic parameters includes:

[0022] A segmentation model based on convolutional networks extracts multiple lane boundary observations from the multiple images;

[0023] Using the vanishing points in the multiple lane boundary observations, the camera's pitch and yaw angles are calculated;

[0024] Based on the consistency of lane width, the camera's roll angle and height are calculated using the camera's pitch and yaw angles.

[0025] A second aspect of this application provides a camera extrinsic parameter calibration device, comprising:

[0026] The acquisition module is used to acquire multiple images captured by the camera;

[0027] The setting module is used to set the first camera extrinsic parameters of the multiple images, and to make the first camera extrinsic parameters and the true values ​​of the camera extrinsic parameters of each of the multiple images follow a normal distribution.

[0028] The first calculation module is used to calculate the first loss function for each of the multiple images based on the first camera extrinsic parameters of each image.

[0029] The second calculation module is used to combine the first loss function of each image with the preset second camera extrinsic parameters to calculate the second loss function of the multiple images;

[0030] An iterative module is used to change the value of the preset second camera extrinsic parameter, iteratively solve the second loss function until the second loss function of the multiple images is not greater than a preset threshold, and use the corresponding preset second camera extrinsic parameter as the target camera extrinsic parameter.

[0031] A third aspect of this application provides an electronic device, comprising:

[0032] Processor; and

[0033] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0034] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0035] The technical solution provided in this application can include the following beneficial effects: This application embodiment acquires multiple images captured by a camera and sets the first camera extrinsic parameters of the multiple images to have a normal distribution between their true values, adapting to the actual driving scenario of a vehicle. Then, based on the first camera extrinsic parameters of the multiple images, a first loss function is calculated for each image. Next, combining the first loss function of each image with a preset second camera extrinsic parameter, a second loss function for the multiple images is calculated. Finally, the value of the preset second camera extrinsic parameter is changed, and the second loss function of the multiple images is iteratively solved until the second loss function of the multiple images is no greater than a preset threshold. The corresponding preset second camera extrinsic parameter is then used as the target camera extrinsic parameter. In this way, the extrinsic parameters between multiple images are constrained by the second loss function, minimizing the difference between the extrinsic parameters of each image and the target camera extrinsic parameter. The obtained target camera extrinsic parameter approximates the true value and can be used as the camera extrinsic parameter for any image captured by the camera while the vehicle is in motion. Furthermore, as the number of images increases, the accuracy of the camera extrinsic parameter also improves, reducing the impact of noise when selecting images.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0037] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0038] Figure 1 This is a first flowchart illustrating the camera extrinsic parameter calibration method according to an embodiment of this application;

[0039] Figure 2 This is a second flowchart illustrating the camera extrinsic parameter calibration method shown in the embodiments of this application;

[0040] Figure 3 This is a schematic diagram of the camera extrinsic parameter calibration device shown in the embodiments of this application;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0042] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0043] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0044] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0045] In related technologies, when calibrating camera extrinsics, the camera extrinsics obtained from the previous image are generally used as the initial values ​​for the extrinsics of the current image to calculate the camera extrinsics for the current image.

[0046] However, in autonomous driving, the camera moves due to the vehicle's vibrations, and the extrinsic parameters of the images captured by the camera are different at different times. The camera extrinsic parameters solved by the above scheme are only applicable to the current frame image and do not approximate the true value, and are not applicable to every image captured by the camera.

[0047] To address the aforementioned issues, this application provides a camera extrinsic parameter calibration method that can use target extrinsic parameters as camera extrinsic parameters for any image captured by a camera while the vehicle is in motion, and the target extrinsic parameters closely approximate the true value with high accuracy.

[0048] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0049] Figure 1 This is a schematic diagram of the first process of the camera extrinsic parameter calibration method shown in the embodiments of this application.

[0050] See Figure 1 A method for calibrating camera extrinsic parameters, comprising steps S1 to S5:

[0051] Step S1: Acquire multiple images captured by the camera.

[0052] It should be noted that multiple images captured by a camera can originate from the camera's video stream data. For example, a specific frame from the camera's video stream can be selected as the image captured by the camera. "Camera" refers to a device capable of recording video; for example, a camera could be a car camera or a mobile phone camera.

[0053] This system can periodically acquire multiple images captured by cameras while the vehicle is in motion. For example, the camera can capture images every 3 seconds, 5 seconds, or 7 seconds while the vehicle is in motion and upload the captured images to the server.

[0054] Step S2: Set the first camera extrinsic parameters for each of the multiple images, wherein the first camera extrinsic parameters and the true values ​​of the camera extrinsic parameters for the multiple images are normally distributed.

[0055] During vehicle movement, the vehicle's offset data within each cycle follows a normal distribution with the vehicle's true position when it is in a preset direction. For example, if the vehicle is scheduled to travel in a preset direction of 0 degrees due north, but due to vehicle vibration, its position within each cycle will not be in the preset direction, but will deviate. Experience shows that the vehicle's offset data within one cycle is equivalent to a normal distribution of the vehicle's true position. For example, at the current moment, the vehicle might be 1 degree to the right of 0 degrees due north; at the next moment, it might be 1 degree to the left of 0 degrees due north; at the next moment, it might be 2 degrees to the right of 0 degrees due north; and so on. Ultimately, the vehicle's offset data and its true position will follow a normal distribution. Therefore, the camera's position will also change with the vehicle's offset, ensuring that the first camera extrinsic parameters and their true values ​​in multiple images captured by the camera also follow a normal distribution.

[0056] Preferably, the true values ​​of the camera's extrinsic parameters include the camera's yaw angle, pitch angle, roll angle, and altitude in the world coordinate system.

[0057] When converting from the world coordinate system to the camera coordinate system, the angles through which an object rotates around the three coordinate axes of the camera coordinate system are the camera's attitude angles, such as yaw, pitch, and roll.

[0058] Step S3: Calculate the first loss function for each image based on the first camera extrinsic parameters of each of the multiple images.

[0059] Multiple lane boundary observations can be extracted from multiple images based on a segmentation model using a convolutional network; the camera's pitch and yaw angles can be calculated using the vanishing points among the multiple lane boundary observations; and the camera's roll and altitude can be calculated using the camera's pitch and yaw angles based on the consistency of lane widths.

[0060] Step S4: Combine the first loss function of each image with the preset second camera extrinsic parameters to calculate the second loss function of multiple images.

[0061] The sum of the first loss functions for each image can be calculated based on the first loss function of each image, resulting in the first equation. The absolute value of the difference between the first camera extrinsic parameter and the preset second camera extrinsic parameter for the i-th image can be calculated, and the sum of the absolute values ​​of the difference between the first camera extrinsic parameter and the preset second camera extrinsic parameter for the i-th image can be multiplied by a preset weight coefficient to obtain the second equation. Where i = 1, 2, 3...n, and n is the total number of the multiple images. The first equation and the second equation are summed to calculate the second loss function.

[0062] Step S5: Change the value of the preset second camera extrinsic parameter, iteratively solve the second loss function of multiple images until the second loss function of multiple images is not greater than the preset threshold, and use the corresponding preset second camera extrinsic parameter as the target camera extrinsic parameter.

[0063] The second loss function of multiple images can be iteratively solved using gradient descent until the second loss function of multiple images is no greater than a preset threshold, and the corresponding preset second camera extrinsic parameters are used as the target camera extrinsic parameters.

[0064] This embodiment acquires multiple images captured by a camera and sets the first camera extrinsic parameters of these images to follow a normal distribution, adapting to the actual driving scenario of a vehicle. Based on the first camera extrinsic parameters of the multiple images, a first loss function is calculated for each image. Then, combining the first loss function of each image with a preset second camera extrinsic parameter, a second loss function for the multiple images is calculated. Finally, the value of the preset second camera extrinsic parameter is changed, and the second loss function of the multiple images is iteratively solved until the second loss function of the multiple images is no greater than a preset threshold. The corresponding preset second camera extrinsic parameter is then used as the target camera extrinsic parameter. This constraint on the extrinsic parameters among the multiple images through the second loss function minimizes the difference between the extrinsic parameters of each image and the target camera extrinsic parameter, resulting in a target camera extrinsic parameter that approximates the true value. This target extrinsic parameter can be used as the camera extrinsic parameter for any image captured by the camera while the vehicle is in motion. Furthermore, as the number of images increases, the accuracy of the camera extrinsic parameter also improves, reducing the impact of noise during image selection.

[0065] Figure 2 This is a second flowchart illustrating the camera extrinsic parameter calibration method shown in the embodiments of this application.

[0066] See Figure 2 The camera extrinsic parameter calibration method in this application includes:

[0067] Step S10: Periodically acquire multiple images captured by the camera while the vehicle is in motion.

[0068] It should be noted that multiple images captured by a camera can originate from the camera's video stream data. For example, a specific frame from the camera's video stream can be selected as the image captured by the camera. "Camera" refers to a device capable of recording video; for example, a camera could be a car camera or a mobile phone camera.

[0069] This system can periodically acquire multiple images captured by cameras while the vehicle is in motion. For example, the camera can capture images every 3 seconds, 5 seconds, or 7 seconds while the vehicle is in motion and upload the captured images to the server.

[0070] Step S20: Set the first camera extrinsic parameters for each of the multiple images, wherein the first camera extrinsic parameters for each of the multiple images and the true values ​​of the camera extrinsic parameters are normally distributed.

[0071] During vehicle movement, the vehicle's offset data within each cycle follows a normal distribution with the vehicle's true position when it is in a preset direction. For example, if the vehicle is scheduled to travel in a preset direction of 0 degrees due north, but due to vehicle vibration, its position within each cycle will not be in the preset direction, but will deviate. Experience shows that the vehicle's offset data within one cycle is equivalent to a normal distribution of the vehicle's true position. For example, at the current moment, the vehicle might be 1 degree to the right of 0 degrees due north; at the next moment, it might be 1 degree to the left of 0 degrees due north; at the next moment, it might be 2 degrees to the right of 0 degrees due north; and so on. Ultimately, the vehicle's offset data and its true position will follow a normal distribution. Therefore, the camera's position will also change with the vehicle's offset, ensuring that the first camera extrinsic parameters and their true values ​​in multiple images captured by the camera also follow a normal distribution.

[0072] Preferably, the true values ​​of the camera's extrinsic parameters include the camera's yaw angle, pitch angle, roll angle, and altitude in the world coordinate system.

[0073] When converting from the world coordinate system to the camera coordinate system, the angles through which the object rotates around the three coordinate axes of the camera coordinate system are the camera's attitude angles, such as yaw, pitch, and roll.

[0074] Step S30: The segmentation model based on convolutional networks extracts multiple lane boundary observations from multiple images.

[0075] Step S31: Calculate the camera's pitch and yaw angles using the vanishing points from multiple lane boundary observations.

[0076] Step S32: Based on the consistency of lane width, calculate the camera's roll angle and height using the camera's pitch and yaw angles.

[0077] Assume the camera intrinsic parameters are known, the road surface is flat, all lane boundaries on the road surface are parallel, and the lane width remains constant.

[0078] First, a convolutional network-based segmentation model is used to extract lane boundary observations from multiple input images. Since the vanishing point (VP) of parallel lane boundaries depends only on pitch and yaw angles and is invariant to changes in roll angle and camera height, a VP can be found from a set of parallel lane boundaries, and the VP is used to estimate the camera's pitch and yaw angles. Then, the pitch and yaw angles are transformed into a rotational relationship between the camera and the VP of the parallel lane boundaries on the road surface, and the roll angle and camera height are calculated.

[0079] Let C and W represent the camera coordinate system and the world coordinate system, respectively. The z-axis of the W world coordinate system is defined as the direction of VP, i.e., VD (vanishing point direction). Then, the pitch and yaw angles can be defined as the angles between the camera's forward direction and VD.

[0080] Step S40: Calculate the sum of the first loss functions for each image based on the first loss function of each image, and obtain the first equation.

[0081] The first equation is: .

[0082] In the formula: Let yaw angle be the angle of the camera corresponding to the i-th image; Let be the camera's pitch angle corresponding to the i-th image; Let be the roll angle of the camera corresponding to the i-th image; Let be the camera height corresponding to the i-th image; n is the total number of images.

[0083] in addition, , , and The values ​​are normally distributed with respect to their corresponding true values.

[0084] Step S41: Calculate the absolute value of the difference between the first camera extrinsic parameter and the preset second camera extrinsic parameter of the i-th image, and multiply the sum of the absolute values ​​of the difference between the first camera extrinsic parameter and the preset second camera extrinsic parameter of the i-th image by a preset weight coefficient to obtain the second equation; where i = 1, 2, 3...n, and n is the total number of the multiple images.

[0085] The second equation is: .

[0086] In the formula: w is the weighting coefficient; Let yaw angle be the angle of the camera corresponding to the i-th image; Let be the camera's pitch angle corresponding to the i-th image; Let be the roll angle of the camera corresponding to the i-th image; Let the height be the camera height corresponding to the i-th image; To preset the yaw angle in the extrinsic parameters of the second camera; To preset the pitch angle in the extrinsic parameters of the second camera; To preset the roll angle in the extrinsic parameters of the second camera; The camera height is preset in the extrinsic parameters of the second camera.

[0087] Step S42: Summing the first equation and the second equation, the second loss function for multiple images is calculated.

[0088] Specifically, the second loss function Loss for multiple images is:

[0089]

[0090] In the formula: Let yaw angle be the angle of the camera corresponding to the i-th image; Let be the camera's pitch angle corresponding to the i-th image; Let be the roll angle of the camera corresponding to the i-th image; The height of the camera corresponding to the i-th image is w; w is the weighting coefficient. To preset the yaw angle in the extrinsic parameters of the second camera; To preset the pitch angle in the extrinsic parameters of the second camera; To preset the roll angle in the extrinsic parameters of the second camera; The camera height is preset in the extrinsic parameters of the second camera.

[0091] Step S50: Iteratively solve the second loss function of multiple images using the gradient descent method until the second loss function of multiple images is not greater than a preset threshold, and use the corresponding preset second camera extrinsic parameters as the target camera extrinsic parameters.

[0092] The preset extrinsic parameters of the second camera are obtained through gradient descent. , , Different values ​​are assigned, and different second loss functions (Loss) are calculated iteratively until the second loss function is no greater than a preset threshold. Then, the corresponding second camera extrinsic parameters are adjusted. , , As an external parameter of the target camera.

[0093] This application embodiment constrains the extrinsic parameters between multiple images using a second loss function Loss, minimizing the difference between the extrinsic parameters of each image and the target camera's extrinsic parameters, thus obtaining target camera extrinsic parameters that approximate the true value. Furthermore, as the number of images increases, the accuracy of the camera's extrinsic parameters also improves, reducing the impact of noise when selecting images.

[0094] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a camera extrinsic parameter calibration device, electronic device, and corresponding embodiments.

[0095] Figure 3 This is a schematic diagram of the camera extrinsic parameter calibration device shown in the embodiments of this application.

[0096] See Figure 3A camera extrinsic parameter calibration device includes an acquisition module 30, a setting module 31, a first calculation module 32, a second calculation module 33, and an iteration module 34.

[0097] The acquisition module 30 is used to acquire multiple images captured by the camera.

[0098] Multiple images captured by a camera can originate from video stream data captured by the camera. For example, a specific frame can be selected from the video stream data captured by the camera as the captured image. "Camera" refers to a device capable of recording video; for example, a camera can be a car camera or a mobile phone camera.

[0099] This system can periodically acquire multiple images captured by cameras while the vehicle is in motion. For example, the camera can capture images every 3 seconds, 5 seconds, or 7 seconds while the vehicle is in motion and upload the captured images to the server.

[0100] The setting module 31 is used to set the first camera extrinsic parameters for each of the multiple images, wherein the first camera extrinsic parameters and the true values ​​of the camera extrinsic parameters for the multiple images are normally distributed.

[0101] During vehicle movement, the vehicle's offset data within each cycle follows a normal distribution with the vehicle's true position when it is in a preset direction. For example, if the vehicle is scheduled to travel in a preset direction of 0 degrees due north, but due to vehicle vibration, its position within each cycle will not be in the preset direction, but will deviate. Experience shows that the vehicle's offset data within one cycle is equivalent to a normal distribution of the vehicle's true position. For example, at the current moment, the vehicle might be 1 degree to the right of 0 degrees due north; at the next moment, it might be 1 degree to the left of 0 degrees due north; at the next moment, it might be 2 degrees to the right of 0 degrees due north; and so on. Ultimately, the vehicle's offset data and its true position will follow a normal distribution. Therefore, the camera's position will also change with the vehicle's offset, ensuring that the first camera extrinsic parameters and their true values ​​in multiple images captured by the camera also follow a normal distribution.

[0102] Preferably, the true values ​​of the camera's extrinsic parameters include the camera's yaw angle, pitch angle, roll angle, and altitude in the world coordinate system.

[0103] When converting from the world coordinate system to the camera coordinate system, the angles through which an object rotates around the three coordinate axes of the camera coordinate system are the camera's attitude angles, such as yaw, pitch, and roll.

[0104] The first calculation module 32 is used to calculate the first loss function for each image based on the first camera extrinsic parameters of each of the multiple images.

[0105] Specifically, the first calculation module 32 is used to extract multiple lane line boundary observations from multiple images using a segmentation model based on a convolutional network; calculate the camera's pitch angle and yaw angle using the vanishing points in the multiple lane line boundary observations; and calculate the camera's roll angle and height using the camera's pitch angle and yaw angle based on the consistency of lane width.

[0106] The second calculation module 33 is used to combine the first loss function of each image with the preset second camera extrinsic parameters to calculate the second loss function of multiple images.

[0107] The second calculation module 33 can calculate the sum of the first loss functions of each image based on the first loss function of each image, and obtain the first equation; calculate the difference between the first camera extrinsic parameters and the preset second camera extrinsic parameters of the i-th image, and multiply the difference between the first camera extrinsic parameters and the preset second camera extrinsic parameters of the i-th image by a preset weight coefficient to obtain the second equation; sum the first equation and the second equation to calculate the second loss function.

[0108] The iteration module 34 is used to change the value of the preset second camera extrinsic parameter, iteratively solve the second loss function of multiple images until the second loss function of multiple images is not greater than a preset threshold, and use the corresponding preset second camera extrinsic parameter as the target camera extrinsic parameter.

[0109] The second loss function of multiple images can be iteratively solved using gradient descent until the second loss function of multiple images is no greater than a preset threshold, and the corresponding preset second camera extrinsic parameters are used as the target camera extrinsic parameters.

[0110] This embodiment acquires multiple images captured by a camera and sets the first camera extrinsic parameters of these images to follow a normal distribution, adapting to the actual driving scenario of a vehicle. Based on the first camera extrinsic parameters of the multiple images, a first loss function is calculated for each image. Then, combining the first loss function of each image with a preset second camera extrinsic parameter, a second loss function for the multiple images is calculated. Finally, the value of the preset second camera extrinsic parameter is changed, and the second loss function of the multiple images is iteratively solved until the second loss function of the multiple images is no greater than a preset threshold. The corresponding preset second camera extrinsic parameter is then used as the target camera extrinsic parameter. This constraint on the extrinsic parameters of multiple images through the second loss function minimizes the difference between the extrinsic parameters of each image and the target camera extrinsic parameter, resulting in a target camera extrinsic parameter that approximates the true value. This target extrinsic parameter can be used as the camera extrinsic parameter for any image captured by the camera while the vehicle is in motion. Furthermore, as the number of images increases, the accuracy of the camera extrinsic parameter also improves, reducing the impact of noise during image selection.

[0111] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0112] Figure 4 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0113] See Figure 4 The electronic device 400 includes a memory 410 and a processor 420.

[0114] The processor 420 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0115] Memory 410 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 420 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 410 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 410 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0116] The memory 410 stores executable code, which, when processed by the processor 420, can cause the processor 420 to execute part or all of the methods described above.

[0117] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0118] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0119] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for calibrating camera extrinsic parameters, characterized in that, include: Acquire multiple images captured by the camera; Each of the plurality of images is given a first camera extrinsic parameter, wherein the first camera extrinsic parameter of each of the plurality of images and the true value of the camera extrinsic parameter are normally distributed; Calculate the first loss function for each image based on the first camera extrinsic parameters of each of the multiple images; By combining the first loss function of each image and the preset second camera extrinsic parameters, a second loss function for the multiple images is calculated; it includes: Based on the first loss function of each image, the sum of the first loss functions of each image is calculated, and the first equation is obtained; the first equation is: In the formula: Let yaw angle be the angle of the camera corresponding to the i-th image; Let be the camera's pitch angle corresponding to the i-th image; Let be the roll angle of the camera corresponding to the i-th image; Let be the camera height corresponding to the i-th image; n is the total number of images. Calculate the absolute value of the difference between the first camera extrinsic parameters and the preset second camera extrinsic parameters for the i-th image, and multiply the sum of the absolute values ​​of the differences between the first camera extrinsic parameters and the preset second camera extrinsic parameters for the i-th image by a preset weighting coefficient to obtain the second equation; where i = 1, 2, 3...n; the second equation is: In the formula: w is the weighting coefficient; To preset the yaw angle in the extrinsic parameters of the second camera; To preset the pitch angle in the extrinsic parameters of the second camera; To preset the roll angle in the extrinsic parameters of the second camera; The camera height is preset in the extrinsic parameters of the second camera; Summing the first equation and the second equation yields the second loss function for the multiple images; Change the value of the preset second camera extrinsic parameter, iteratively solve the second loss function of the multiple images until the second loss function of the multiple images is not greater than a preset threshold, and use the corresponding preset second camera extrinsic parameter as the target camera extrinsic parameter.

2. The method according to claim 1, characterized in that, The second loss function of the multiple images is solved iteratively using the gradient descent method.

3. The method according to claim 1, characterized in that, The acquisition of multiple images captured by the camera includes: The system periodically acquires multiple images captured by the camera while the vehicle is in motion.

4. The method according to claim 3, characterized in that, The vehicle's offset data in each cycle during driving follows a normal distribution with the true value of the vehicle's position when it is in the preset direction.

5. The method according to claim 1, characterized in that, The true values ​​of the camera's extrinsic parameters include the camera's yaw angle, pitch angle, roll angle, and altitude in the world coordinate system.

6. The method according to claim 1, characterized in that, The step of calculating the first loss function for each image based on the first camera extrinsic parameters of each of the multiple images includes: A segmentation model based on convolutional networks extracts multiple lane boundary observations from the multiple images; Using the vanishing points in the multiple lane boundary observations, the camera's pitch and yaw angles are calculated; Based on the consistency of lane width, the camera's roll angle and height are calculated using the camera's pitch and yaw angles.

7. A camera extrinsic parameter calibration device, characterized in that, include: The acquisition module is used to acquire multiple images captured by the camera; The setting module is used to set the first camera extrinsic parameters of the multiple images, wherein the first camera extrinsic parameters of each of the multiple images and the true values ​​of the camera extrinsic parameters are normally distributed. The first calculation module is used to calculate the first loss function for each of the multiple images based on the first camera extrinsic parameters of each image. The second calculation module is used to calculate the second loss function of the multiple images by combining the first loss function of each image and the preset second camera extrinsic parameters; it is used for: Based on the first loss function of each image, the sum of the first loss functions of each image is calculated, and the first equation is obtained; the first equation is: In the formula: Let yaw angle be the angle of the camera corresponding to the i-th image; Let be the camera's pitch angle corresponding to the i-th image; Let be the roll angle of the camera corresponding to the i-th image; Let be the camera height corresponding to the i-th image; n is the total number of images. Calculate the absolute value of the difference between the first camera extrinsic parameters and the preset second camera extrinsic parameters for the i-th image, and multiply the sum of the absolute values ​​of the differences between the first camera extrinsic parameters and the preset second camera extrinsic parameters for the i-th image by a preset weighting coefficient to obtain the second equation; where i = 1, 2, 3...n; the second equation is: In the formula: w is the weighting coefficient; To preset the yaw angle in the extrinsic parameters of the second camera; To preset the pitch angle in the extrinsic parameters of the second camera; To preset the roll angle in the extrinsic parameters of the second camera; The camera height is preset in the extrinsic parameters of the second camera; Summing the first equation and the second equation yields the second loss function for the multiple images; An iterative module is used to change the value of the preset second camera extrinsic parameter, iteratively solve the second loss function until the second loss function of the multiple images is not greater than a preset threshold, and use the corresponding preset second camera extrinsic parameter as the target camera extrinsic parameter.

8. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-6.

Images