External Parameter Calibration Method, Device, Vehicle, and Storage Medium of In-Vehicle Surround View Camera

Abstract

To provide an external parameter calibration method of an on-vehicle surround view camera for improving real-time properties and versatility of calibration of external parameters of an on-vehicle surround view camera, and to provide a device, a vehicle, and a storage medium.SOLUTION: A method in a vehicle having an on-vehicle surround view camera with first to fourth cameras includes: determination of external parameters of the first and second cameras according to visual odometer information; determination of initial values of first to fourth position postures; and determination of external parameters of the third and fourth cameras according to them. Accordingly, it becomes possible to avoid a problem of a large amount of calculation and long calculation time, improve real-time properties for calibration of external parameters, and increase precision of calibration of external parameters by calculating the respective external parameter initial values of the four cameras and then jointly optimizing the external parameter initial values of the four cameras.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of autonomous driving, and in particular, to an external parameter calibration method and apparatus for an in-vehicle surround view camera, a vehicle, and a storage medium.

Background Art

[0002] In an intelligent driving assistance system, since the calibration of an in-vehicle surround view camera is the basis for most intelligent sensing functions, the stability and accuracy of the surround view camera calibration directly affect the accuracy of the visual sensing function. Camera calibration may be divided into internal parameter calibration and external parameter calibration. The internal parameters of a camera are the inherent parameters of the camera, and the external parameters of a camera are the parameters representing the position and attitude relationship of the camera coordinate system with respect to the vehicle body coordinate system. The vehicle body coordinate system is a coordinate system established with the vehicle itself as the reference system, and may be considered as a coordinate system for explaining the relative position and attitude relationship between the objects around the vehicle and the vehicle. The camera coordinate system is a three-dimensional rectangular coordinate system established with the focus center of the camera as the origin and the optical axis as the Z axis.

[0003] Currently, the external parameter calibration method of an in-vehicle surround view camera may be divided into the following three types. The first is a camera calibration method based on a specific location. Basically, this method adopts a specific calibration template placed in the scene and performs calibration by extracting the feature points of the calibration template. However, such a method requires a large investment in the construction cost of the location and can only be applied to an environment where the vehicle is stationary. The second is a camera calibration method based on the vanishing point. It calculates the vanishing point in the image by using the parallel information in the environment and then calculates the external parameters of the camera according to the vanishing point. However, this method depends on special markers with parallel information, such as lanes, and the application scenarios are limited. The third is a camera calibration method based on the fusion of the scene map. It calculates the relative position and orientation between the cameras by fusing the map information. However, in order to establish the environmental map, it is necessary for each camera to activate the odometer. The calculation requirements are high, the time taken is long, and it is disadvantageous for real-time online calibration.

[0004] Therefore, how to improve the real-time performance and versatility of the external parameter calibration of the in-vehicle surround view camera has become an urgent technical problem to be solved currently.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Embodiments of the present invention provide an external parameter calibration method, device, vehicle, and storage medium for an in-vehicle surround view camera to improve the real-time performance and versatility of the external parameter calibration of the in-vehicle surround view camera.

Means for Solving the Problems

[0006] According to one aspect of the embodiments of the present invention, Applied to a vehicle equipped with an in-vehicle surround view camera including a first camera, a second camera, a third camera, and a fourth camera, wherein the position of the first camera is adjacent to the positions of the third camera and the fourth camera, and the position of the second camera is adjacent to the positions of the third camera and the fourth camera, Determining the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera; Determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera; Determining a first target position and orientation initial value, a second target position and orientation initial value, a third target position and orientation initial value, and a fourth target position and orientation initial value according to the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value; Determining the external parameters of the third camera and the external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value, mi including The external parameter is a parameter representing the positional and postural relationship of the vehicle body coordinate system with respect to the camera coordinate system. An external parameter calibration method for an in-vehicle surround view camera is provided.

[0007] According to another aspect of the embodiments of the present invention, Disposed on a vehicle equipped with an in-vehicle surround view camera including a first camera, a second camera, a third camera, and a fourth camera, wherein the position of the first camera is adjacent to the positions of the third camera and the fourth camera, and the position of the second camera is adjacent to the positions of the third camera and the fourth camera, A first external parameter determination module for determining the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera; A position and orientation determination module for determining a first initial position and orientation value, a second initial position and orientation value, a third initial position and orientation value, and a fourth initial position and orientation value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera; An initial value determination module for determining a first target position and orientation initial value, a second target position and orientation initial value, a third target position and orientation initial value, and a fourth target position and orientation initial value according to the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value; A second external parameter determination module for determining the external parameters of the third camera and the external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value. An external parameter calibration device for an in-vehicle surround view camera is provided.

[0008] According to another aspect of the embodiments of the present invention, An in-vehicle surround view camera; At least one processor; A memory communicatively connected to the at least one processor, and a vehicle is provided, wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the external parameter calibration method of the in-vehicle surround view camera described in any one of the embodiments of the present invention.

[0009] ​According to another aspect of an embodiment of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement an external parameter calibration method for an in-vehicle surround view camera described in any embodiment of the present invention during execution.

Advantages of the Invention

[0010] The technical aspect of the embodiment of the present invention determines the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera, and determines the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera, and determines the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value according to the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value, and determines the external parameters of the third camera and the external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value. This technical aspect first determines the corresponding external parameters according to the visual odometer information of the first camera and the second camera, and then determines the external parameters of the third camera and the fourth camera according to the external parameters of the first camera and the second camera, thereby avoiding the problem of large calculation and long time caused by all four cameras performing visual odometer calculations to determine the external parameters, improving the real-time performance of external parameter calibration, and this aspect can be applied to external parameter calibration in any scenario, avoiding the problem of relying on specific markers on the road, and improving the versatility of external parameter calibration.

[0011] The content described in this section is not intended to identify the core or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will be more easily understood from the following description.

Brief Description of the Drawings

[0012] To more clearly illustrate the technical aspects in the embodiments of the present invention, the drawings necessary for use in the description of the embodiments will be briefly introduced below. However, the drawings described below are only some embodiments of the present invention, and it is obvious that those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0014] To help those skilled in the art better understand the aspects of the present invention, the technical aspects in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. However, it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present invention.

[0015] In addition, the terms "first", "second", etc. in the specification and claims of the present invention, as well as in the above drawings, do not necessarily need to be used to explain a specific order or sequence, but are for distinguishing similar objects. The data used in this way can be replaced when appropriate, so it should be understood that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. Also, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a series of steps or unit processes, methods, systems, products or devices are included, and are not necessarily limited to those steps or units clearly listed, and may also include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0016] Embodiment 1 FIG. 1 is a schematic diagram of the flow of an external parameter calibration method for an in-vehicle surround view camera according to Embodiment 1 of the present invention. The method is applicable when calibrating the external parameters of the in-vehicle surround view camera. The method can be executed by an external parameter calibration device for the in-vehicle surround view camera. Among them, the device can be realized by software and / or hardware, and generally, it is integrated in a vehicle equipped with an in-vehicle surround view camera including a first camera, a second camera, a third camera, and a fourth camera. The position of the first camera is adjacent to the positions of the third camera and the fourth camera, and the position of the second camera is adjacent to the positions of the third camera and the fourth camera. The position can be understood as the position where the camera is located.

[0017] As shown in FIG. 1, the external parameter calibration method of the in-vehicle surround view camera according to Embodiment 1 of the present invention includes the following steps.

[0018] In S110, according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera, the external parameters of the first camera and the external parameters of the second camera are determined.

[0019] In this embodiment, the first camera, the second camera, the third camera, and the fourth camera can be understood as cameras used for capturing images in the front view, back view, left view, and right view directions of the vehicle with the vehicle body as a reference. Among them, the setting of the direction corresponding to each camera is not limited, but it is necessary to ensure that the position of the first camera is adjacent to the position of the third camera and the position of the fourth camera, and the position of the second camera is adjacent to the position of the third camera and the position of the fourth camera. That is, if the first camera is the front view camera, the third camera may be the left view camera or the right view camera adjacent to the position of the front view camera. Correspondingly, if the second camera is the back view camera, the fourth camera may be the left view camera or the right view camera adjacent to the second camera. It is necessary to ensure that the positions corresponding to each camera are different. For example, it can be understood that both cameras cannot be the left view camera.

[0020] The first visual odometer information is considered to be the visual odometer information corresponding to the first camera. The second visual odometer information is considered to be the visual odometer information corresponding to the second camera.

[0021] The vehicle body odometer information can be understood as information related to the vehicle body odometer of the vehicle. The vehicle body odometer can be understood as representing information on changes in the moving position of the vehicle based on the vehicle body. The visual odometer information is information related to the visual odometer of the corresponding camera. The visual odometer can be understood as representing information on changes in the moving position of the camera based on the camera and information such as map points. Here, the specific content of the vehicle body odometer information and the visual odometer information is not specifically limited. For example, the vehicle body odometer information may include information such as vehicle speed and vehicle body movement trajectory, and the visual odometer information may include information such as map points and camera movement trajectory. The vehicle body movement trajectory can be understood as information representing the movement trajectory of the vehicle when it moves. The camera movement trajectory can be understood as information representing the moving trajectory of the camera when it moves.

[0022] Here, it is not specifically limited how to determine the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera. For example, first, obtain the vehicle body odometer information of the vehicle by methods such as the real-time kinematic (RTK) method, the inertial measurement unit (IMU) method, and the wheel speed meter method. Then, based on the images captured by the first camera and the second camera, obtain the first visual odometer information and the second visual odometer information by corresponding visual odometer calculation methods. Finally, based on the vehicle body odometer information, the first visual odometer information, and the second visual odometer information, obtain the external parameters of the first camera and the external parameters of the second camera by corresponding external parameter algorithms (for example, an algorithm based on a preset hand-eye calibration principle).

[0023] In S120, according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera, determine a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value.

[0024] In this embodiment, the first position and orientation initial value can be understood as an initial value representing the relative position and orientation between the first camera and the third camera. The second position and orientation initial value can be understood as an initial value representing the relative position and orientation between the first camera and the fourth camera. The third position and orientation initial value is an initial value representing the relative position and orientation between the second camera and the third camera. The fourth position and orientation initial value is an initial value representing the relative position and orientation between the second camera and the fourth camera.

[0025] Here, it is not specifically limited as to how to determine the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera. For example, first, according to the vehicle body odometer information, the map points in each visual odometer information, and the image acquisition ranges of each camera, the map points in the overlapping area of the acquisition ranges between the first camera and the third camera, the map points in the overlapping area of the acquisition ranges between the first camera and the fourth camera, the map points in the overlapping area of the acquisition ranges between the second camera and the third camera, and the map points in the overlapping area of the acquisition ranges between the second camera and the fourth camera are determined. Then, according to the external parameters of the first camera and the second camera, the determined map points are first converted into the camera coordinate system and then into the vehicle body coordinate system to obtain the map points in the vehicle body coordinate system. According to the map points in the vehicle body coordinate system and the images captured by each camera, the corresponding initial external parameter determination algorithm is used to determine the initial external parameter values corresponding to the third camera and the fourth camera. Finally, according to the external parameters of the first camera and the second camera, and the initial external parameter values corresponding to the third camera and the fourth camera, the corresponding position and orientation initial value determination algorithm may be used to determine the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value.

[0026] In S130, according to the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value, a first target initial position and orientation value, a second target initial position and orientation value, a third target initial position and orientation value, and a fourth target initial position and orientation value are determined.

[0027] In this embodiment, the initial target position and orientation can be understood as the initial position and orientation for determining the external parameters of the third camera and the fourth camera. Here, without specifically limiting how to determine the first initial target position and orientation, the second initial target position and orientation, the third initial target position and orientation, and the fourth initial target position and orientation, for example, the first initial position and orientation, the second initial position and orientation, the third initial position and orientation, and the fourth initial position and orientation can be directly determined as the first initial target position and orientation, the second initial target position and orientation, the third initial target position and orientation, and the fourth initial target position and orientation respectively, or the first initial position and orientation, the second initial position and orientation, the third initial position and orientation, and the fourth initial position and orientation can be optimized by a corresponding optimization algorithm, and the optimized initial position and orientation values can be determined as the respective initial target position and orientation values.

[0028] In S140, according to the external parameters of the first camera, the external parameters of the second camera, the first initial target position and orientation, the second initial target position and orientation, the third initial target position and orientation, and the fourth initial target position and orientation, determine the external parameters of the third camera and the external parameters of the fourth camera.

[0029] TIFF0007698095000001.tif76170

[0030] TIFF0007698095000002.tif49170

[0031] FIG. 2 is a realization schematic diagram constructed in a closed-loop relationship according to Embodiment 1 of the present invention. As shown in FIG. 2, taking the front view camera, the left view camera, and the vehicle body as an example, between the front view camera (i.e., the first camera) and the left view camera (i.e., the third camera), the positions are adjacent, and between the front view camera, the left view camera, and the vehicle body, a closed-loop relationship is formed.

[0032] TIFF0007698095000003.tif27170

[0033] The external parameter calibration method of the in-vehicle surround view camera according to Embodiment 1 of the present invention determines the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera, and determines the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera. According to the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value are determined. According to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value, the external parameters of the third camera and the external parameters of the fourth camera are determined. This technical aspect first determines the corresponding external parameters according to the visual odometer information of the first camera and the second camera, and then determines the external parameters of the third camera and the fourth camera according to the external parameters of the first camera and the second camera, thereby avoiding the problem of large calculation and long time caused by all four cameras performing visual odometer calculations to determine the external parameters, improving the real-time performance of external parameter calibration, and this technical aspect is applicable to external parameter calibration in any scenario, avoiding the problem of relying on specific markers on the road, and improving the versatility of external parameter calibration.

[0034] Embodiment 2 FIG. 4 is a schematic diagram of the flow of the external parameter calibration method for an in-vehicle surround view camera according to Embodiment 2 of the present invention. Embodiment 2 is refined based on each of the above embodiments. In this embodiment, the process of determining the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera, and the process of determining the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera are specifically described. Note that for the technical details not described in detail in this embodiment, any of the above embodiments can be referred to. As shown in FIG. 4, the method includes the following.

[0035] In S210, vehicle body odometer information and images captured by each camera are acquired.

[0036] In this embodiment, the vehicle can acquire the vehicle body odometer information of the vehicle by means of a corresponding algorithm. The vehicle can acquire the images captured by each camera in the in-vehicle surround view camera.

[0037] In S220, based on the images captured by the first camera and the second camera, the first visual odometer information and the second visual odometer information are determined.

[0038] In this embodiment, the vehicle can obtain the corresponding first visual odometer information by means of a corresponding algorithm based on the image captured by the first camera, and the vehicle can obtain the corresponding second visual odometer information by means of a corresponding algorithm based on the image captured by the second camera, and here, it is not limited thereto.

[0039] In S230, based on the vehicle body odometer information and the determined visual odometer information, the external parameters of the first camera and the external parameters of the second camera are determined by a predetermined hand-eye calibration algorithm.

[0040] In this embodiment, the predetermined hand-eye calibration algorithm can be understood as an external parameter calculation method based on a preset hand-eye calibration principle. The vehicle can obtain the external parameters of the first camera by a predetermined hand-eye calibration algorithm based on the vehicle body odometer information and the first visual odometer information, and the vehicle can obtain the external parameters of the second camera by a predetermined hand-eye calibration algorithm based on the vehicle body odometer information and the second visual odometer information. Here, it is not limited thereto.

[0041] In S240, according to the vehicle body odometer information, the first map point in the first visual odometer information, the second map point in the second visual odometer information, and the capture range of each camera, the first valid map point, the second valid map point, the third valid map point, and the fourth valid map point are determined.

[0042] In this embodiment, the map point can be understood as a three-dimensional point corresponding to the feature point of the image captured by the camera. Correspondingly, the first map point can be understood as the map point captured by the first camera. The second map point can be understood as the map point captured by the second camera.

[0043] The first valid map point is the map point in the overlapping area between the first camera and the third camera. The second valid map point is the map point in the overlapping area between the first camera and the fourth camera. The third valid map point is the map point in the overlapping area between the second camera and the third camera. The fourth valid map point is the map point in the overlapping area between the second camera and the fourth camera.

[0044] Here, without specifically limiting how to determine the first valid map point, the second valid map point, the third valid map point, and the fourth valid map point. For example, first determine a scale factor according to the vehicle body odometer information and the visual odometer information, proportionally transform the first map point and the second map point based on the scale factor, obtain the first real-world map point and the second real-world map point in the real-world map corresponding to the scale factor, and then determine the overlap area between the first camera and the third camera (i.e., the first overlap area), the overlap area between the first camera and the fourth camera (i.e., the second overlap area), the overlap area between the second camera and the third camera (i.e., the third overlap area), and the overlap area between the second camera and the fourth camera (i.e., the fourth overlap area). Finally, determine the map point in the first overlap area of the first real-world map point as the first valid map point, determine the map point in the second overlap area of the first real-world map point as the second valid map point, determine the map point in the third overlap area of the second real-world map point as the third valid map point, and determine the map point in the fourth overlap area of the second real-world map point as the fourth valid map point.

[0045] In S250, according to the positions and postures of the first visual odometer information and the second visual odometer information, convert the first valid map point and the second valid map point into the first camera coordinate system to obtain the first coordinate point and the second coordinate point, and convert the third valid map point and the fourth valid map point into the second camera coordinate system to obtain the third coordinate point and the fourth coordinate point.

[0046] In this embodiment, the first coordinate point can be understood as the coordinate point obtained by converting the first valid map point into the first camera coordinate system. The second coordinate point can be understood as the coordinate point obtained by converting the second valid map point into the first camera coordinate system. The third coordinate point can be understood as the coordinate point obtained by converting the third valid map point into the second camera coordinate system. The fourth coordinate point can be understood as the coordinate point obtained by converting the fourth valid map point into the second camera coordinate system.

[0047] In S260, according to the external parameters of the first camera and the external parameters of the second camera, the first coordinate point, the second coordinate point, the third coordinate point, and the fourth coordinate point are respectively converted into the vehicle body coordinate system to obtain the first vehicle body coordinate point, the second vehicle body coordinate point, the third vehicle body coordinate point, and the fourth vehicle body coordinate point.

[0048] In this embodiment, the first vehicle body coordinate point can be understood as the point where the first valid map point is converted into the vehicle body coordinate system. The second vehicle body coordinate point can be understood as the point where the second valid map point is converted into the vehicle body coordinate system. The third vehicle body coordinate point can be understood as the point where the third valid map point is converted into the vehicle body coordinate system. The fourth vehicle body coordinate point can be understood as the point where the fourth valid map point is converted into the vehicle body coordinate system.

[0049] The external parameters of the first camera represent the position and orientation relationship of the first camera coordinate system in the vehicle body coordinate system. After obtaining the inverse of the external parameters of the first camera to obtain the position and orientation relationship of the vehicle body coordinate system in the first camera coordinate system, based on this position and orientation relationship, the first coordinate point can be converted into a point in the vehicle body coordinate system, that is, the first vehicle body coordinate point, and the second coordinate point can be converted into a point in the vehicle body coordinate system, that is, the second vehicle body coordinate point. Correspondingly, the external parameters of the second camera represent the position and orientation relationship of the second camera coordinate system in the vehicle body coordinate system. After obtaining the inverse of the external parameters of the second camera to obtain the position and orientation relationship of the vehicle body coordinate system in the second camera coordinate system, based on this position and orientation relationship, the third coordinate point can be converted into a point in the vehicle body coordinate system, that is, the third vehicle body coordinate point, and the fourth coordinate point can be converted into a point in the vehicle body coordinate system, that is, the fourth vehicle body coordinate point.

[0050] In S270, according to the images captured by each camera, the first vehicle body coordinate point, the second vehicle body coordinate point, the third vehicle body coordinate point, and the fourth vehicle body coordinate point, the initial values of the first target external parameters and the second target external parameters of the third camera, and the initial values of the third target external parameters and the fourth target external parameters of the fourth camera are determined.

[0051] In this embodiment, the first target external parameter initial value and the second target external parameter initial value can be understood as the optimized first external parameter initial value and the second external parameter initial value, and the first external parameter initial value and the second external parameter initial value can be understood as two initial values representing the external parameters of the third camera. The third target external parameter initial value and the fourth target external parameter initial value can be understood as the optimized third external parameter initial value and the fourth external parameter initial value, and the third external parameter initial value and the fourth external parameter initial value can be understood as two initial values representing the external parameters of the fourth camera.

[0052] Here, there is no specific limitation on how to determine each target external parameter initial value. For example, the images captured by each camera are processed by a corresponding algorithm. The corresponding feature points in the images of the third camera for the first vehicle body coordinate points obtained by the first camera and the third camera through feature point matching, the corresponding feature points in the images of the fourth camera for the second vehicle body coordinate points obtained by the first camera and the fourth camera through feature point matching, the corresponding feature points in the images of the third camera for the third vehicle body coordinate points obtained by the second camera and the third camera through feature point matching, and the corresponding feature points in the images of the fourth camera for the fourth vehicle body coordinate points obtained by the second camera and the fourth camera through feature point matching are extracted. The feature points can represent pixel points with prominent features. Then, according to each extracted feature point and each determined vehicle body coordinate point, the initial external parameters of the third camera and the initial external parameters of the fourth camera are determined by a corresponding algorithm. Finally, the initial external parameters of the third camera and the initial external parameters of the fourth camera are optimized by a preset optimization algorithm to obtain the first target external parameter initial value and the second target external parameter initial value of the third camera, as well as the third target external parameter initial value and the fourth target external parameter initial value of the fourth camera.

[0053] In S280, according to the first target external parameter initial value, the second target external parameter initial value, the third target external parameter initial value, the fourth target external parameter initial value, the external parameters of the first camera, and the external parameters of the second camera, determine the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value.

[0054] In this embodiment, first, according to the first target external parameter initial value, the second target external parameter initial value, the third target external parameter initial value, the fourth target external parameter initial value, the external parameters of the first camera, and the external parameters of the second camera, the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value can be obtained by corresponding algorithms.

[0055] In S290, according to the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value, determine the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value.

[0056] In S2100, according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value, determine the external parameters of the third camera and the external parameters of the fourth camera.

[0057] The external parameter calibration method of the in-vehicle surround view camera according to Embodiment 2 of the present invention embodies the process of determining the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera, and the process of determining the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera. Using this method, the external parameters of the other two cameras can be determined by the external parameters of the two cameras determined previously, avoiding the problems of large calculation and long time caused by the operation of the visual odometer calculation of the cameras for all four cameras, effectively improving the efficiency of external parameter calibration, and guaranteeing the real-time performance of external parameter calibration.

[0058] Preferably, determining the first valid map point, the second valid map point, the third valid map point, and the fourth valid map point according to the vehicle body odometer information, the first map point in the first visual odometer information, the second map point in the second visual odometer information, and the acquisition range of each camera is determining a scale factor, which is a parameter representing the ratio between the scale of the real world and the scale of the three-dimensional map constructed by the camera, based on the vehicle body movement trajectory information in the vehicle body odometer information and the camera movement trajectory information in the determined visual odometer information, proportionally transforming the first map point and the second map point based on the scale factor to obtain the first real map point at the scale of the real world of the first map point and the second real map point at the scale of the real world of the second map point, According to the acquisition range of each camera, determine a first overlap area which is the overlap area of the acquisition ranges between the first camera and the third camera, a second overlap area which is the overlap area of the acquisition ranges between the first camera and the fourth camera, a third overlap area which is the overlap area of the acquisition ranges between the second camera and the third camera, and a fourth overlap area which is the overlap area of the acquisition ranges between the second camera and the fourth camera, including determining the map points in the first overlap area of the first real-world map points as the first valid map points, determining the map points in the second overlap area of the first real-world map points as the second valid map points, determining the map points in the third overlap area of the second real-world map points as the third valid map points, and determining the map points in the fourth overlap area of the second real-world map points as the fourth valid map points.

[0059] In this embodiment, the scale of the real world can be understood as the scale in the real world. The scale of the three-dimensional map constructed by the camera can be understood as the scale for constructing the three-dimensional map. The vehicle body movement trajectory information can be understood as the information representing the vehicle body movement trajectory. The camera movement trajectory information can be understood as the information representing the camera movement trajectory. Based on the vehicle body movement trajectory information in the vehicle body odometer information and the camera movement trajectory information in the determined visual odometer information, the vehicle can obtain the corresponding scale factor by means of the corresponding algorithm.

[0060] Preferably, according to the images captured by each camera, the first vehicle body coordinate point, the second vehicle body coordinate point, the third vehicle body coordinate point, and the fourth vehicle body coordinate point, determining the first initial target external parameter value and the second initial target external parameter value of the third camera, and the third initial target external parameter value and the fourth initial target external parameter value of the fourth camera is According to the images captured by each camera, it includes determining a first target feature point which is the corresponding feature point in the image of the third camera of the first vehicle body coordinate points obtained by the first camera and the third camera through feature point matching, a second target feature point which is the corresponding feature point in the image of the fourth camera of the second vehicle body coordinate points obtained by the first camera and the fourth camera through feature point matching, a third target feature point which is the corresponding feature point in the image of the third camera of the third vehicle body coordinate points obtained by the second camera and the third camera through feature point matching, and a fourth target feature point which is the corresponding feature point in the image of the fourth camera of the fourth vehicle body coordinate points obtained by the second camera and the fourth camera through feature point matching. In this embodiment, there is no specific limitation on how to determine the first target feature point, the second target feature point, the third target feature point, and the fourth target feature point according to the images captured by each camera. The feature point of the first camera is represented as a two-dimensional feature point corresponding to the map point in the first visual odometer information. The feature point of the second camera is represented as a two-dimensional feature point corresponding to the map point in the second visual odometer information. The feature points of the first camera and the third camera, and the feature points of the fourth camera are matched by a feature matching algorithm to determine the first target feature point and the second target feature point. The feature points of the second camera and the third camera, and the feature points of the fourth camera are matched by a feature matching algorithm to determine the third target feature point and the fourth target feature point. Here, the feature matching algorithm is not specifically limited.

[0061] According to each vehicle body coordinate point and each target feature point, the initial values of the first external parameters and the second external parameters of the third camera, and the initial values of the third external parameters and the fourth external parameters of the fourth camera are determined by the Perspective-n-Point (PNP) method. In this embodiment, the external parameter initial value can be understood as the initial value of the external parameter. Here, without specific limitation on how to determine the external parameter initial value of the third camera and the external parameter initial value of the fourth camera according to each vehicle body coordinate point and each target feature point. For example, by using a corresponding algorithm (such as the PNP algorithm), the first external parameter initial value of the third camera is obtained based on the first vehicle body coordinate point and the first target feature point, the second external parameter initial value of the third camera is obtained based on the second vehicle body coordinate point and the second target feature point, the first external parameter initial value of the fourth camera is obtained based on the third vehicle body coordinate point and the third target feature point, and the second external parameter initial value of the fourth camera is obtained based on the fourth vehicle body coordinate point and the fourth target feature point.

[0062] For each of the third camera and the fourth camera, for each numerical value in the predetermined region of the camera, update the external parameter initial value of the camera to a numerical value, project the vehicle body coordinate point corresponding to the camera onto the image coordinate system of the camera to obtain the corresponding two-dimensional coordinate point, determine the photometric error between the corresponding two-dimensional coordinate point of the camera and the target feature point, and the predetermined region is related to the external parameter initial value of the corresponding camera. In this embodiment, the predetermined region is related to the external parameter initial value of the corresponding camera. Here, there is no limitation on the predetermined region. For example, it may be a predetermined range region determined with the external parameter initial value as the intermediate value, or it may be a predetermined range region determined with the external parameter initial value as the left interval endpoint. The image coordinate system can be understood as a coordinate system established by taking the center of the image plane as the coordinate origin and making the X-axis and the Y-axis parallel to the two vertical sides of the image plane respectively. The image coordinate system is considered as a coordinate system indicating the position of the pixel in the image in physical units (such as millimeters). The photometric error can be understood as the difference value between the photometric value of the two-dimensional coordinate point and the photometric value of the target feature point. Each numerical value in the predetermined region can obtain a corresponding photometric error.

[0063] Among the corresponding photometric errors of the camera, determine the numerical value corresponding to the minimum photometric error as the corresponding target external parameter initial value of the camera.

[0064] In this embodiment, the minimum photometric error can be understood as the photometric error with the smallest value.

[0065] Preferably, determining the first initial target position and orientation value, the second initial target position and orientation value, the third initial target position and orientation value, and the fourth initial target position and orientation value according to the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value includes determining the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value as the first initial target position and orientation value, the second initial target position and orientation value, the third initial target position and orientation value, and the fourth initial target position and orientation value.

[0066] Preferably, determining the first initial target position and orientation value, the second initial target position and orientation value, the third initial target position and orientation value, and the fourth initial target position and orientation value according to the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value if the first photometric error value at the first initial position and orientation value, the second photometric error value at the second initial position and orientation value, the third photometric error value at the third initial position and orientation value, and the fourth photometric error value at the fourth initial position and orientation value do not satisfy a first predetermined condition, updating each initial position and orientation value based on a first predetermined step size until each photometric error value at the updated initial position and orientation value satisfies the first predetermined condition, and determining each initial position and orientation value as each initial target position and orientation value, wherein the first photometric error value is an error value between the photometric value of the first projection point and the photometric value of the first target feature point, and the first projection point is a point obtained by projecting the first valid map point of the first camera at the first initial position and orientation value onto the image coordinate system of the third camera, the second photometric error value is an error value between the photometric value of the second projection point and the photometric value of the second target feature point, and the second projection point is a point obtained by projecting the second valid map point of the first camera at the second initial position and orientation value onto the image coordinate system of the fourth camera, The third photometric error value is the error value between the photometric value of the third projection point and the photometric value of the third target feature point. The third projection point is a point obtained by projecting the third valid map point of the second camera in the third initial position and orientation onto the image coordinate system of the third camera. The fourth photometric error value is the error value between the photometric value of the fourth projection point and the photometric value of the fourth target feature point. The fourth projection point is a point obtained by projecting the fourth valid map point of the second camera in the fourth initial position and orientation onto the image coordinate system of the fourth camera. The first predetermined condition is that the first photometric error value reaches the first threshold, the second photometric error value reaches the second threshold, the third photometric error value reaches the third threshold, and the fourth photometric error value reaches the fourth threshold.

[0067] In this embodiment, the vehicle can obtain a first initial position and orientation by a predetermined position and orientation algorithm according to the external parameters of the first camera and the initial value of the target external parameters of the third camera, and can obtain a second initial position and orientation by a predetermined position and orientation algorithm according to the external parameters of the first camera and the initial value of the target external parameters of the fourth camera. Correspondingly, the vehicle can obtain a third initial position and orientation by a predetermined position and orientation algorithm according to the external parameters of the second camera and the initial value of the target external parameters of the third camera, and can obtain a fourth initial position and orientation by a predetermined position and orientation algorithm according to the external parameters of the second camera and the initial value of the target external parameters of the fourth camera.

[0068] In this embodiment, the first threshold can be understood as the preset first threshold, the second threshold can be understood as the preset second threshold, the third threshold can be understood as the preset third threshold, and the fourth threshold can be understood as the preset fourth threshold. Here, the first threshold, the second threshold, the third threshold, and the fourth threshold are not specifically limited. When the first photometric error value reaches the first threshold, it is considered that the first photometric error value is smaller than or equal to the first threshold. Correspondingly, when the second photometric error value reaches the second threshold, it is considered that the second photometric error value is smaller than or equal to the second threshold. When the third photometric error value reaches the third threshold, it is considered that the third photometric error value is smaller than or equal to the third threshold. When the fourth photometric error value reaches the fourth threshold, it is considered that the fourth photometric error value is smaller than or equal to the fourth threshold.

[0069] The first predetermined step size can be understood as the preset step size, and here it is not limited thereto. Here, there is no specific limitation on how to update the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and / or the fourth initial position and orientation value based on the first predetermined step size. For example, the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and / or the fourth initial position and orientation value can be increased by the first predetermined step size, or the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and / or the fourth initial position and orientation value can be decreased by the first predetermined step size. After the first initial position and orientation value is updated, the relative position and orientation between the first camera and the third camera change, and the corresponding first photometric error changes. Correspondingly, after the second initial position and orientation value is updated, the relative position and orientation between the first camera and the fourth camera change, and the corresponding second photometric error changes. It can be understood by analogy based on this.

[0070] Preferably, after determining the external parameters of the third camera and the external parameters of the fourth camera, Determining the error index by the following formula with the external parameters of each camera is further included, TIFF0007698095000004.tif9170 Among them, E rr is an error index, which is a parameter index for measuring the optimization status of the external parameters of each camera. E1 is the reprojection error value obtained by projecting the first valid map point of the first camera onto the image coordinate system of the third camera. E2 is the reprojection error value obtained by projecting the third valid map point of the second camera onto the image coordinate system of the third camera. E3 is the reprojection error value obtained by projecting the second valid map point of the first camera onto the image coordinate system of the fourth camera. E4 is the reprojection error value obtained by projecting the fourth valid map point of the second camera onto the image coordinate system of the fourth camera. E5 is the vehicle body odometer information and the first visual odometer of the first camera information is the hand-eye calibration error value determined based on them. E6 is the vehicle body odometer information and the second visual odometer of the second camera information is the hand-eye calibration error value determined based on them. In this embodiment, the reprojection error can be understood as the pixel difference between the point obtained by projecting the valid map point in the camera coordinate system onto the image coordinate system and the true point in the image coordinate system. The hand-eye calibration error value can be understood as the odometer error value between the vehicle body odometer and the visual odometer calculated based on the hand-eye calibration principle.

[0071] TIFF0007698095000005.tif57170

[0072] TIFF0007698095000006.tif180170

[0073] If the error index reaches the error threshold, the external parameters of each camera are determined as the target external parameters of each camera. The target external parameters are the optimized external parameters. If the error index does not reach the error threshold, the external parameters of at least one camera are updated based on the second predetermined step size until the corresponding error index of each camera reaches the error threshold, and the external parameters of each corresponding camera when the error threshold is reached are determined as the target external parameters of each camera.

[0074] In this embodiment, the error threshold can be understood as a threshold for measuring a preset error index, and here, it is not specifically limited. The second predetermined step size can be understood as a preset second step size, and here, it is not specifically limited.

[0075] Here, it is not specifically limited as to how to update the external parameters of at least one camera based on the second predetermined step size. For example, the external parameters of at least one camera may be increased or decreased by the second predetermined step size to obtain new external parameters.

[0076] In this embodiment, after calculating the initial values of the external parameters of the four cameras, the initial values of the external parameters of the four cameras are optimized in a unified manner to further improve the accuracy of external parameter calibration.

[0077] Preferably, after determining the target external parameters of each camera, for each camera, based on the target external parameters of the camera, determining Euler angle information, which is information representing the rotation angle of the camera with respect to the vehicle body coordinate system of the camera, and translation vector information, which is information representing the translation amount of the camera with respect to the vehicle body coordinate system of the camera; judging whether the Euler angle information and the translation vector information of the camera satisfy a second predetermined condition that the Euler angle information is within a first threshold range and the translation vector information is within a second threshold range; if satisfied, determining that the calibration of the target external parameters of the camera is successful; if not satisfied, determining that the calibration of the target external parameters of the camera has failed, are further included.

[0078] In this embodiment, the Euler angle information may include the pitch angle α, yaw angle β, and roll angle σ of the camera. Here, there is no specific limitation on how to determine the Euler angle information and translation vector information of the camera based on the target external parameters of the camera. For example, the target external parameters of the camera may be identified and processed by a corresponding algorithm to obtain the Euler angle information and translation vector information of the camera.

[0079] The first threshold range can be understood as the preset first threshold range, and the second threshold range can be understood as the preset second threshold range. Here, there is no specific limitation on the first threshold range and the second threshold range. For example, the first threshold range may be set as 0° < α < 90°, -90° < β < 90°, -90° < σ < 90°.

[0080] Embodiment 3 FIG. 5 is a schematic structural diagram of an external parameter calibration device for an in-vehicle surround view camera according to Embodiment 3 of the present invention, and the device can be realized by software and / or hardware. As shown in FIG. 5, the device is arranged in a vehicle equipped with an in-vehicle surround view camera including a first camera, a second camera, a third camera, and a fourth camera. The position of the first camera is adjacent to the positions of the third camera and the fourth camera, and the position of the second camera is adjacent to the positions of the third camera and the fourth camera. The device includes: a first external parameter determination module 310 for determining the external parameters of the first camera and the external parameters of the second camera according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera; a position and orientation determination module 320 for determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera; An initial value determination module 330 for determining a first target position and orientation initial value, a second target position and orientation initial value, a third target position and orientation initial value, and a fourth target position and orientation initial value according to the first position and orientation initial value, the second position and orientation initial value, the third position and orientation initial value, and the fourth position and orientation initial value; A second external parameter determination module 340 for determining the external parameters of the third camera and the external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value.

[0081] In this embodiment, according to the vehicle body odometer information of the vehicle, the first visual odometer information of the first camera, and the second visual odometer information of the second camera, the first external parameter determination module determines the external parameters of the first camera and the external parameters of the second camera. According to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera, the position and orientation determination module determines the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value. According to the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value, the initial value determination module determines the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value. According to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value, the second external parameter determination module determines the external parameters of the third camera and the external parameters of the fourth camera. This technical solution first determines the corresponding external parameters according to the visual odometer information of the first camera and the second camera, and then determines the external parameters of the third camera and the fourth camera according to the external parameters of the first camera and the second camera, thereby avoiding the problem of large calculation and long time caused by all four cameras performing visual odometer calculations to determine the external parameters, improving the real-time performance of external parameter calibration. In addition, this technical solution is applicable to external parameter calibration in any scenario, avoiding the problem of relying on specific markers on the road, and improving the versatility of external parameter calibration.

[0082] Preferably, the first external parameter determination module 310 includes an image acquisition unit for acquiring the vehicle body odometer information and the images captured by each camera, An odometer determination unit for determining the first visual odometer information and the second visual odometer information based on the images captured by the first camera and the second camera; An external parameter determination unit for determining the external parameters of the first camera and the external parameters of the second camera by a predetermined hand-eye calibration algorithm based on the vehicle body odometer information and the determined visual odometer information.

[0083] Preferably, the position and orientation determination module 320 A map point determination unit for determining a first valid map point, a second valid map point, a third valid map point, and a fourth valid map point according to the vehicle body odometer information, a first map point in the first visual odometer information, a second map point in the second visual odometer information, and the capture ranges of the respective cameras; A first conversion unit for converting the first valid map point and the second valid map point into a first coordinate point and a second coordinate point by converting them into the first camera coordinate system according to the position and orientation of the first visual odometer information and the second visual odometer information, and converting the third valid map point and the fourth valid map point into a third coordinate point and a fourth coordinate point by converting them into the second camera coordinate system; A second conversion unit for converting the first coordinate point, the second coordinate point, the third coordinate point, and the fourth coordinate point into a first vehicle body coordinate point, a second vehicle body coordinate point, a third vehicle body coordinate point, and a fourth vehicle body coordinate point by converting them into the vehicle body coordinate system according to the external parameters of the first camera and the external parameters of the second camera; An external parameter initial value determination unit for determining a first target external parameter initial value and a second target external parameter initial value of the third camera, and a third target external parameter initial value and a fourth target external parameter initial value of the fourth camera according to the images captured by the respective cameras, the first vehicle body coordinate point, the second vehicle body coordinate point, the third vehicle body coordinate point, and the fourth vehicle body coordinate point; A position and orientation initial value determination unit for determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value according to the first target external parameter initial value, the second target external parameter initial value, the third target external parameter initial value, the fourth target external parameter initial value, the external parameters of the first camera, and the external parameters of the second camera.

[0084] Preferably, the map point determination unit A factor determination subunit for determining a scale factor, which is a parameter representing the ratio between the scale of the real world and the scale of the three-dimensional map constructed by the camera, based on the vehicle body movement trajectory information in the vehicle body odometer information and the camera movement trajectory information in the determined visual odometer information. A conversion subunit for proportionally converting the first map point and the second map point based on the scale factor to obtain a first real map point at the scale of the real world of the first map point and a second real map point at the scale of the real world of the second map point. An area determination unit for determining a first overlap area, which is an overlap area of the acquisition ranges between the first camera and the third camera, a second overlap area, which is an overlap area of the acquisition ranges between the first camera and the fourth camera, a third overlap area, which is an overlap area of the acquisition ranges between the second camera and the third camera, and a fourth overlap area, which is an overlap area of the acquisition ranges between the second camera and the fourth camera, according to the acquisition range of each camera. A map point determination subunit for determining the map point in the first overlap area of the first real map point as the first valid map point, determining the map point in the second overlap area of the first real map point as the second valid map point, determining the map point in the third overlap area of the second real map point as the third valid map point, and determining the map point in the fourth overlap area of the second real map point as the fourth valid map point.

[0085] Preferably, the external parameter initial value determination unit According to the images captured by each camera, a first target feature point that is a corresponding feature point in the image of the third camera of the first vehicle body coordinate point obtained by the first camera and the third camera through feature point matching, a second target feature point that is a corresponding feature point in the image of the fourth camera of the second vehicle body coordinate point obtained by the first camera and the fourth camera through feature point matching, a third target feature point that is a corresponding feature point in the image of the third camera of the third vehicle body coordinate point obtained by the second camera and the third camera through feature point matching, and a fourth target feature point that is a corresponding feature point in the image of the fourth camera of the fourth vehicle body coordinate point obtained by the second camera and the fourth camera through feature point matching, a feature point determination subunit for determining; According to each vehicle body coordinate point and each target feature point, a first initial value determination subunit for determining the first external parameter initial value and the second external parameter initial value of the third camera, and the third external parameter initial value and the fourth external parameter initial value of the fourth camera by the PNP method; For each camera among the third camera and the fourth camera, for each numerical value in a predetermined region of the camera related to the external parameter initial value of the corresponding camera, update the external parameter initial value of the camera to the numerical value, project the corresponding vehicle body coordinate point of the camera into the image coordinate system of the camera to obtain a corresponding two-dimensional coordinate point, and an error determination subunit for determining the photometric error between the corresponding two-dimensional coordinate point of the camera and the target feature point; Among the corresponding photometric errors of the camera, a second initial value determination subunit for determining the numerical value corresponding to the minimum photometric error as the corresponding target external parameter initial value of the camera.

[0086] Preferably, the initial value determination module 330 A first initial value determination unit is provided for determining the first initial position and orientation value, the second initial position and orientation value, the third initial position and orientation value, and the fourth initial position and orientation value as a first target initial position and orientation value, a second target initial position and orientation value, a third target initial position and orientation value, and a fourth target initial position and orientation value.

[0087] Preferably, the initial value determination module 330 If the first photometric error value at the first initial position and orientation value, the second photometric error value at the second initial position and orientation value, the third photometric error value at the third initial position and orientation value, and the fourth photometric error value at the fourth initial position and orientation value do not satisfy a first predetermined condition, each of the initial position and orientation values is updated based on a first predetermined step size until each photometric error value at the updated initial position and orientation values satisfies the first predetermined condition, and a second initial value determination unit is further provided for determining each of the initial position and orientation values as each of the target initial position and orientation values. Among them, the first photometric error value is an error value between the photometric value of the first projection point and the photometric value of the first target feature point, and the first projection point is a point obtained by projecting the first valid map point of the first camera at the first initial position and orientation value onto the image coordinate system of the third camera. The second photometric error value is an error value between the photometric value of the second projection point and the photometric value of the second target feature point, and the second projection point is a point obtained by projecting the second valid map point of the first camera at the second initial position and orientation value onto the image coordinate system of the fourth camera. The third photometric error value is an error value between the photometric value of the third projection point and the photometric value of the third target feature point, and the third projection point is a point obtained by projecting the third valid map point of the second camera at the third initial position and orientation value onto the image coordinate system of the third camera. The fourth photometric error value is an error value between the photometric value of the fourth projection point and the photometric value of the fourth target feature point, and the fourth projection point is a point obtained by projecting the fourth valid map point of the second camera at the fourth initial position and orientation value onto the image coordinate system of the fourth camera. The first predetermined condition is that the first photometric error value reaches a first threshold value, the second photometric error value reaches a second threshold value, the third photometric error value reaches a third threshold value, and the fourth photometric error value reaches a fourth threshold value.

[0088] Preferably, the apparatus After determining the external parameters of the third camera and the external parameters of the fourth camera, an error index is determined by the following mathematical formula using the external parameters of each camera: TIFF0007698095000007.tif8170 Among them, E rr is the error index which is a parameter index for measuring the optimization status of the external parameters of each camera. E1 is the reprojection error value obtained by projecting the first valid map point of the first camera onto the image coordinate system of the third camera. E2 is the reprojection error value obtained by projecting the third valid map point of the second camera onto the image coordinate system of the third camera. E3 is the reprojection error value obtained by projecting the second valid map point of the first camera onto the image coordinate system of the fourth camera. E4 is the reprojection error value obtained by projecting the fourth valid map point of the second camera onto the image coordinate system of the fourth camera. E5 is the vehicle body odometer information and the first visual odometer of the first camera information is the hand-eye calibration error value determined based on the above, and E6 is the vehicle body odometer information and the second visual odometer of the second camera information and an index determination module for the hand-eye calibration error value determined based on the above, If the error index reaches an error threshold value, a third external parameter determination module is used to determine the external parameters of each camera as the target external parameters which are the optimized external parameters of each camera. If the error index does not reach the error threshold value, based on a second predetermined step size, the external parameters of at least one camera are updated until the corresponding error index of each camera reaches the error threshold value, and a fourth external parameter determination module is used to determine the external parameters of the corresponding cameras at the time when the error threshold value is reached as the target external parameters of each camera.

[0089] Preferably, the apparatus After determining the target external parameters of each camera, for each camera, based on the target external parameters of the camera, an information determination module for determining Euler angle information, which is information representing the rotation angle of the camera with respect to the vehicle body coordinate system of the camera, and translation vector information, which is information representing the translation amount of the camera with respect to the vehicle body coordinate system of the camera, A determination module for determining whether the Euler angle information and the translation vector information of the camera satisfy a second predetermined condition that the Euler angle information is within a first threshold range and the translation vector information is within a second threshold range, If satisfied, a first calibration module for determining that the calibration of the target external parameters of the camera has been successful, If not satisfied, a second calibration module for determining that the calibration of the target external parameters of the camera has failed, and further includes.

[0090] The external parameter calibration apparatus for an in-vehicle surround view camera according to an embodiment of the present invention is capable of executing the external parameter calibration method for an in-vehicle surround view camera according to any embodiment of the present invention, and includes a functional module and beneficial effects corresponding to the execution of the method.

[0091] Example 4 FIG. 6 is a schematic structural diagram of a vehicle according to Embodiment 4 of the present invention. As shown in FIG. 6, the vehicle 10 includes an in-vehicle surround view camera 20, at least one processor 11, and a memory communicatively connected to the at least one processor 11, for example, a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by at least one processor. The processor 11 is capable of performing various appropriate operations and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 further stores various programs and data required for the operation of the vehicle 10. The in-vehicle surround view camera 20, the processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0092] Connected to the I / O interface 15 are a plurality of components in the vehicle 10 including an input unit 16 such as a keyboard and a mouse, an output unit 17 such as various types of displays and speakers, a storage unit 18 such as a magnetic disk and an optical disk, and a communication unit 19 such as a network card, a modem, and a wireless communication transceiver. The communication unit 19 enables the vehicle 10 to exchange information / data with other devices via, for example, a computer network of the Internet and / or various telecommunication networks.

[0093] Processor 11 may be a general-purpose and / or dedicated processing component with various processing and computing capabilities. Some examples of Processor 11 include a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that execute algorithms of machine learning models, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc., but are not limited thereto. Processor 11 executes each of the methods and processes described above, for example, the external parameter calibration method of the in-vehicle surround view camera.

[0094] In some embodiments, the external parameter calibration method of the in-vehicle surround view camera can be realized as a computer program, which is tangibly included in a computer-readable storage medium, such as storage unit 18. In some embodiments, the computer program can be loaded and / or installed in vehicle 10 partially or entirely via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by Processor 11, one or more steps of the above-described external parameter calibration method of the in-vehicle surround view camera can be executed. Alternatively, in other embodiments, Processor 11 is configured to execute the external parameter calibration method of the in-vehicle surround view camera by any other suitable means (e.g., by firmware).

[0095] In this specification, various embodiments of the above-described systems and techniques can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs, which can be executed and / or interpreted in a programmable system including at least one programmable processor, the programmable processor being a dedicated or general-purpose programmable processor that can receive data and instructions from a memory system, at least one input device, and at least one output device and transmit the data and instructions to the memory system, the at least one input device, and the at least one output device.

[0096] A computer program for implementing the method of the present invention may be written using any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing apparatus, whereby when the computer program is executed by the processor, the functions / operations defined in the flowchart and / or block diagram are implemented. The computer program may be executed entirely on the device, partially on the device, as a stand-alone software package, partially on the device and partially on a remote device, or entirely on a remote device or server.

[0097] In the context of the present invention, a computer-readable storage medium may be a tangible medium that includes or can store a computer program for use in or in combination with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Optionally, the computer-readable storage medium may be a device-readable signal medium. More specific examples of machine-readable storage media include electrical connections by one or more lines, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical memory devices, magnetic memory devices, or any suitable combination of the foregoing.

[0098] To provide for interaction with a user, the systems and techniques described herein can be implemented on a vehicle that includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the vehicle. Other types of devices can be used to provide for interaction with a user, and for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and input received from the user can be in any form (including voice input, speech input, or tactile input).

[0099] The systems and techniques described herein may be implemented in a computing system that includes a background component (e.g., as a data server), or in a computing system that includes a middleware component (such as an application server), or in a computing system that includes a front-end component (such as a user computer having a graphical user interface or a network browser, and through which a user can interact with embodiments of the systems and techniques described herein), or in a computing system that includes any combination of such background, middleware, or front-end components. The components of the system may be connected to each other by digital data communication in any form or medium (e.g., a communication network). Exemplary communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0100] The computing system may include client terminals and servers. Client terminals and servers are generally located apart from each other and usually interact via a communication network. The relationship between the client terminal and the server is established by execution on corresponding computers and by computer programs having a client terminal-server relationship with each other. The server may be a cloud server, which is also called a cloud computing server or a cloud host, and is a hosting product in a cloud computing service system that solves the defects of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0101] It should be understood that the various forms of flow shown above can be used to rearrange, add, or delete steps. For example, each step described in the present invention may be executed in parallel, sequentially, or in a different order as long as the desired result of the technical aspect according to the present invention can be achieved, and is not limited herein.

[0102] The above specific embodiments do not limit the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention should all be included in the protection scope of the present invention.

Claims

1. The present invention is applied to a vehicle equipped with an in-vehicle surround view camera including a first camera, a second camera, a third camera, and a fourth camera, wherein the position of the first camera is adjacent to the position of the third camera and the position of the fourth camera, the position of the second camera is adjacent to the position of the third camera and the position of the fourth camera, determining external parameters of the first camera and external parameters of the second camera according to body odometer information of the vehicle, first visual odometer information of the first camera, and second visual odometer information of the second camera; determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, the external parameters of the first camera, and the external parameters of the second camera; determining a first target position / posture initial value, a second target position / posture initial value, a third target position / posture initial value, and a fourth target position / posture initial value according to the first position / posture initial value, the second position / posture initial value, the third position / posture initial value, and the fourth position / posture initial value; determining external parameters of the third camera and external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value; The external parameters are parameters that represent a position and attitude relationship of a camera coordinate system to a vehicle body coordinate system. The present invention relates to a method for calibrating external parameters of an in-vehicle surround view camera.

2. determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value in accordance with the vehicle body odometer information, the first visual odometer information, the second visual odometer information, an external parameter of the first camera, and an external parameter of the second camera, determining a first valid map point, a second valid map point, a third valid map point and a fourth valid map point according to the vehicle body odometer information, a first map point in the first visual odometer information, a second map point in the second visual odometer information and a capturing range of each camera; transforming the first and second effective map points into a first camera coordinate system to obtain first and second coordinate points, and transforming the third and fourth effective map points into a second camera coordinate system to obtain third and fourth coordinate points, according to the positions and attitudes of the first and second visual odometer information; transforming the first coordinate point, the second coordinate point, the third coordinate point, and the fourth coordinate point into a vehicle body coordinate system according to an external parameter of the first camera and an external parameter of the second camera, respectively, to obtain a first vehicle body coordinate point, a second vehicle body coordinate point, a third vehicle body coordinate point, and a fourth vehicle body coordinate point; determining a first target external parameter initial value and a second target external parameter initial value of the third camera, and a third target external parameter initial value and a fourth target external parameter initial value of the fourth camera, according to the images captured by each camera, the first vehicle body coordinate point, the second vehicle body coordinate point, the third vehicle body coordinate point, and the fourth vehicle body coordinate point; determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value according to the first target external parameter initial value, the second target external parameter initial value, the third target external parameter initial value, the fourth target external parameter initial value, the external parameters of the first camera, and the external parameters of the second camera; 2. The method of claim 1 .

3. determining a first target position / posture initial value, a second target position / posture initial value, a third target position / posture initial value, and a fourth target position / posture initial value according to the first position / posture initial value, the second position / posture initial value, the third position / posture initial value, and the fourth position / posture initial value; determining the first position / posture initial value, the second position / posture initial value, the third position / posture initial value, and the fourth position / posture initial value as a first target position / posture initial value, a second target position / posture initial value, a third target position / posture initial value, and a fourth target position / posture initial value; 2. The method of claim 1 .

4. determining a first target position / posture initial value, a second target position / posture initial value, a third target position / posture initial value, and a fourth target position / posture initial value according to the first position / posture initial value, the second position / posture initial value, the third position / posture initial value, and the fourth position / posture initial value; if a first luminance error value in the first position / posture initial value, a second luminance error value in the second position / posture initial value, a third luminance error value in the third position / posture initial value, and a fourth luminance error value in the fourth position / posture initial value do not satisfy a first predetermined condition, updating each of the position / posture initial values ​​based on a first predetermined step size until each luminance error value in each of the updated position / posture initial values ​​satisfies a first predetermined condition, and determining each of the position / posture initial values ​​as each of the target position / posture initial values; the first luminosity error value is an error value between a luminosity of a first projection point and a luminosity of a first target feature point, the first projection point being a point obtained by projecting a first effective map point of the first camera onto an image coordinate system of the third camera at a first initial position and orientation value; the second luminosity error value is an error value between a luminosity of a second projected point and a luminosity of a second target feature point, and the second projected point is a point obtained by projecting a second effective map point of the first camera onto an image coordinate system of the fourth camera at a second initial position and orientation value; the third luminosity error value is an error value between a luminosity of a third projection point and a luminosity of a third target feature point, the third projection point being a point obtained by projecting a third effective map point of the second camera onto an image coordinate system of the third camera at a third initial position and orientation value; the fourth luminosity error value is an error value between a luminosity of a fourth projected point and a luminosity of a fourth target feature point, and the fourth projected point is a point obtained by projecting a fourth effective map point of the second camera onto an image coordinate system of the fourth camera at a fourth initial position and orientation value; the first predetermined condition is that the first luminosity error value reaches a first threshold, the second luminosity error value reaches a second threshold, the third luminosity error value reaches a third threshold, and the fourth luminosity error value reaches a fourth threshold.

3. The method of claim 2 .

5. determining a first valid map point, a second valid map point, a third valid map point, and a fourth valid map point according to the vehicle body odometer information, the first map point in the first visual odometer information, the second map point in the second visual odometer information, and the capturing range of each camera; Determining a scale factor, which is a parameter representing a ratio between a scale of the real world and a scale of a three-dimensional map constructed by a camera, based on the vehicle body motion trajectory information in the vehicle body odometer information and the camera motion trajectory information in the determined visual odometer information; proportionally transforming the first map point and the second map point based on the scale factor to obtain a first real map point at a real world scale for the first map point and a second real map point at a real world map scale for the second map point; determining a first overlap area, which is an overlap area of ​​the collection range between the first camera and the third camera, a second overlap area, which is an overlap area of ​​the collection range between the first camera and the fourth camera, a third overlap area, which is an overlap area of ​​the collection range between the second camera and the third camera, and a fourth overlap area, which is an overlap area of ​​the collection range between the second camera and the fourth camera, according to the collection range of each camera; determining a map point in the first overlap area of ​​the first real map point as a first valid map point, determining a map point in the second overlap area of ​​the first real map point as a second valid map point, determining a map point in the third overlap area of ​​the second real map point as a third valid map point, and determining a map point in the fourth overlap area of ​​the second real map point as a fourth valid map point.

3. The method of claim 2 .

6. determining a first target external parameter initial value and a second target external parameter initial value of the third camera and a third target external parameter initial value and a fourth target external parameter initial value of the fourth camera according to the images captured by each camera, the first vehicle body coordinate point, the second vehicle body coordinate point, the third vehicle body coordinate point, and the fourth vehicle body coordinate point; According to the images captured by each camera, determining a first target feature point which is a corresponding feature point in the image of the third camera at a first body coordinate point obtained by the first camera and the third camera through feature point matching, a second target feature point which is a corresponding feature point in the image of the fourth camera at a second body coordinate point obtained by the first camera and the fourth camera through feature point matching, a third target feature point which is a corresponding feature point in the image of the third camera at a third body coordinate point obtained by the second camera and the third camera through feature point matching, and a fourth target feature point which is a corresponding feature point in the image of the fourth camera at a fourth body coordinate point obtained by the second camera and the fourth camera through feature point matching; determining a first external parameter initial value and a second external parameter initial value of the third camera, and a third external parameter initial value and a fourth external parameter initial value of the fourth camera according to each vehicle body coordinate point and each target feature point by a perspective n-point positioning PNP method; For each of the third and fourth cameras, for each numerical value in a predetermined area of ​​the camera that is related to an initial external parameter value of the corresponding camera, update the initial external parameter value of the camera to the numerical value, project the body coordinate point corresponding to the camera into an image coordinate system of the camera to obtain a corresponding two-dimensional coordinate point, and determine a luminance error between the corresponding two-dimensional coordinate point of the camera and a target feature point; determining a numerical value corresponding to a minimum luminance error among the luminance errors corresponding to the camera as a corresponding target external parameter initial value of the camera; 3. The method of claim 2 .

7. After determining the external parameters of the third camera and the external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value, For the external parameters of each camera, the error index is determined by the following formula: E ｒｒ is the error index, which is a parameter index that evaluates the optimization status of the external parameters of each camera, and E 1 is a reprojection error value obtained by projecting the first effective map point of the first camera onto the image coordinate system of the third camera, and E 2 is a reprojection error value obtained by projecting a third valid map point of the second camera onto the image coordinate system of the third camera, and E 3 is a reprojection error value obtained by projecting the second effective map point of the first camera onto the image coordinate system of the fourth camera, and E 4 is a reprojection error value obtained by projecting the fourth valid map point of the second camera onto the image coordinate system of the fourth camera, and E 5 is a hand-eye calibration error value determined based on the vehicle odometer information and the first visual odometer information of the first camera, and E 6 is a hand-eye calibration error value determined based on the vehicle body odometer information and the second visual odometer information of the second camera; If the error index reaches an error threshold, determine the extrinsic parameters of each camera as target extrinsic parameters of each camera, and the target extrinsic parameters are the extrinsic parameters after optimization; If the error indicator does not reach the error threshold, updating the extrinsic parameters of at least one camera based on a second predetermined step size until the corresponding error indicator of each camera reaches the error threshold, and determining the corresponding extrinsic parameters of each camera as target extrinsic parameters of each camera when the error threshold is reached.

2. The method of claim 1 .

8. After determining the target external parameters of each camera, determining, for each camera, Euler angle information representing a rotation angle of the camera relative to a vehicle body coordinate system and translation vector information representing an amount of translation of the camera relative to the vehicle body coordinate system, based on the target external parameters of the camera; determining whether the Euler angle information and the translation vector information of the camera satisfy a second predetermined condition, that is, the Euler angle information is within a first threshold range and the translation vector information is within a second threshold range; If so, determining that the target extrinsic parameter calibration of the camera is successful; If not, determining that the target extrinsic parameter calibration of the camera has failed.

7. The method of claim 6.

9. The vehicle is provided with an in-vehicle surround view camera having a first camera, a second camera, a third camera, and a fourth camera, the position of the first camera is adjacent to the position of the third camera and the position of the fourth camera, the position of the second camera is adjacent to the position of the third camera and the position of the fourth camera, a first external parameter determination module for determining external parameters of the first camera and external parameters of the second camera according to body odometer information of the vehicle, first visual odometer information of the first camera, and second visual odometer information of the second camera; a position and orientation determination module for determining a first position and orientation initial value, a second position and orientation initial value, a third position and orientation initial value, and a fourth position and orientation initial value according to the vehicle body odometer information, the first visual odometer information, the second visual odometer information, an external parameter of the first camera, and an external parameter of the second camera; an initial value determination module for determining a first target position / posture initial value, a second target position / posture initial value, a third target position / posture initial value, and a fourth target position / posture initial value according to the first position / posture initial value, the second position / posture initial value, the third position / posture initial value, and the fourth position / posture initial value; a second external parameter determination module for determining external parameters of the third camera and external parameters of the fourth camera according to the external parameters of the first camera, the external parameters of the second camera, the first target position and orientation initial value, the second target position and orientation initial value, the third target position and orientation initial value, and the fourth target position and orientation initial value; The present invention relates to an external parameter calibration device for an in-vehicle surround view camera.

10. An in-vehicle surround view camera, At least one processor; a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the external parameter calibration method for an in-vehicle surround view camera according to any one of claims 1 to 8. A vehicle characterized by:

11. A processor is stored with computer instructions for implementing the external parameter calibration method for an in-vehicle surround view camera according to any one of claims 1 to 8 when executed. A computer-readable storage medium comprising:

Images