A Method, Device and Surround-View Aerial View Acquisition System for Determining Extrinsic Parameters of a Surround-View Camera

Abstract

The embodiments of the present application provide a method and apparatus for determining the external parameters of a surround camera and a surround bird's-eye view acquisition system. Based on the image data collected by the first surround camera and the second surround camera, a first virtual image and a second virtual image are generated. According to the initial external parameters of the first surround camera and the second surround camera and the initial pose of the movable object, the projection points of the spatial points in the ground co-visible area in the first virtual image and the first projection point in the second virtual image are determined; the initial external parameters are adjusted according to the first difference between the pixel values of the first projection point and the second projection point to obtain the corrected external parameters of the first surround camera and the second surround camera. The corrected external parameters determined based on the present application can more accurately stitch the images collected by the surround camera.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular, to a method and apparatus for determining the extrinsic parameters of a surround camera and a surround bird's-eye view acquisition system. Background Art

[0002] To accurately obtain the surrounding environment of a target object, such as a vehicle, an automated guided robot, etc., and reasonably control the movement of the target object according to the surrounding environment, surround cameras can be set around the target object. Exemplarily, as Figure 1a shown. Based on the image data collected by each surround camera, bird's-eye views in various directions of the target object are generated, and by stitching these bird's-eye views, a surround bird's-eye view that can effectively reflect the surrounding situation of the target object can be generated. Exemplarily, as Figure 1b shown.

[0003] When stitching the bird's-eye views in various directions, the stitching is based on the poses of the surround cameras. If the poses of the surround cameras cannot be accurately determined, it may cause the bird's-eye views in various directions to not be stitched accurately. Exemplarily, as Figure 1c shown, where the zebra in the surround bird's-eye view appears broken.

[0004] Therefore, how to accurately determine the extrinsic parameters of each surround camera has become a technical problem to be solved urgently. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method for determining the extrinsic parameters of a surround camera to accurately determine the extrinsic parameters of each surround camera. The specific technical solution is as follows:

[0006] According to a first aspect of the embodiments of the present application, a method for determining the extrinsic parameters of a surround camera is provided. The method includes:

[0007] Based on the image data collected by a first surround camera, a first virtual image obtained by a first virtual camera collecting a ground common view area is generated, where the first virtual camera has the same position as the first surround camera and faces the ground common view area, the ground common view area is an overlapping area between the field of view of the first surround camera and the field of view of a second surround camera, and the first surround camera and the second surround camera are disposed on a movable object;

[0008] Based on the image data collected by the second surround camera, a second virtual image obtained by a second virtual camera collecting the ground common view area is generated, where the second virtual camera has the same position as the second surround camera and faces the ground common view area;

[0009] Determine the first projection point of the spatial point in the ground common view area in the first virtual image and the second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object;

[0010] Adjust the initial extrinsic parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

[0011] In a possible embodiment, the method further includes:

[0012] Generate a bird's-eye view of the ground common view area based on the image data collected by the first surround camera;

[0013] Determine the pixel points whose gradients satisfy the preset gradient condition in the bird's-eye view as target pixel points, where the preset gradient condition includes: the gradient is greater than a preset gradient threshold;

[0014] The determining the first projection point of the spatial point in the ground common view area in the first virtual image and the second projection point of the first projection point in the second virtual image includes:

[0015] Project the target pixel points onto the first virtual image to obtain the first projection points; and project the first projection points onto the second virtual image to obtain the second projection points.

[0016] In a possible embodiment, the adjusting the initial extrinsic parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object includes:

[0017] Adjust the initial extrinsic parameters and the initial pose according to the first difference between the pixel values of the first projection point and the second projection point to obtain the calibrated pose of the movable object and the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object in the calibrated pose.

[0018] In a possible embodiment, the method further includes:

[0019] Determine the third projection point of the first projection point in the second virtual image according to the calibrated extrinsic parameters and the calibrated pose;

[0020] Until the second difference between the pixel values of the first projection point and the third projection point satisfies a preset convergence condition, adjust the correction external parameters and the correction pose according to the second difference to obtain new correction external parameters and a new correction pose, and return to execute the step of determining the third projection point of the first projection point in the second virtual image according to the correction external parameters and the correction pose.

[0021] In a possible embodiment, the method further includes:

[0022] According to the initial external parameters of the first surround-view camera and the second surround-view camera, respectively determine the matching degrees between the positions of the first surround-view camera and the second surround-view camera and the prior position conditions;

[0023] The adjusting the initial external parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the correction external parameters of the first surround-view camera and the second surround-view camera relative to the movable object includes:

[0024] Adjust the initial external parameters according to the first difference between the pixel values of the first projection point and the second projection point and the matching degree to obtain the correction external parameters of the first surround-view camera and the second surround-view camera relative to the movable object.

[0025] In a possible embodiment, the adjusting the initial external parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the correction external parameters of the first surround-view camera and the second surround-view camera relative to the movable object includes:

[0026] Determine the gradient descent direction of the loss function with the initial external parameters as the independent variable and the first difference as the dependent variable;

[0027] Adjust the initial external parameters along the gradient descent direction to obtain the correction external parameters of the first surround-view camera and the second surround-view camera relative to the movable object.

[0028] According to the second aspect of the embodiments of the present application, a surround-view bird's-eye view acquisition system is provided. The system includes a first surround-view camera, a second surround-view camera, and a processor. There is an overlapping ground co-view area between the field of view of the first surround-view camera and the field of view of the second surround-view camera;

[0029] The first surround-view camera and the second surround-view camera are disposed on the movable object, and the first surround-view camera and the second surround-view camera are used to collect image data at different orientations of the movable object;

[0030] The processor is configured to generate a first virtual image obtained by a first virtual camera capturing a ground co - visible area based on the image data collected by the first surround camera; generate a second virtual image obtained by a second virtual camera capturing the ground co - visible area based on the image data collected by the second surround camera; determine a first projection point of a spatial point in the ground co - visible area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object, and adjust the initial extrinsic parameters according to a first difference in pixel values between the first projection point and the second projection point to obtain corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object; wherein, the position of the first virtual camera is the same as that of the first surround camera and faces the ground co - visible area, and the position of the second virtual camera is the same as that of the second surround camera and faces the ground co - visible area;

[0031] The processor is further configured to generate bird's - eye views at different orientations of the movable object based on the correction parameters and the corrected extrinsic parameters of the first surround camera and the second surround camera; and stitch the bird's - eye views to obtain a surround bird's - eye view of the movable object.

[0032] According to a third aspect of the embodiments of the present application, there is provided a device for determining the extrinsic parameters of a surround camera, the device includes:

[0033] A first generation module, configured to generate a first virtual image obtained by a first virtual camera capturing a ground co - visible area based on the image data collected by the first surround camera, wherein the position of the first virtual camera is the same as that of the first surround camera and faces the ground co - visible area, the ground co - visible area is an overlapping area between the field of view of the first surround camera and the field of view of the second surround camera, and the first surround camera and the second surround camera are disposed on a movable object;

[0034] A second generation module, configured to generate a second virtual image obtained by a second virtual camera capturing the ground co - visible area based on the image data collected by the second surround camera, wherein the position of the second virtual camera is the same as that of the second surround camera and faces the ground co - visible area;

[0035] A determination module, configured to determine a first projection point of a spatial point in the ground co - visible area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object;

[0036] An adjustment module, configured to adjust the initial external parameters according to a first difference between pixel values of the first projection point and the second projection point, so as to obtain corrected external parameters of the first panoramic camera and the second panoramic camera relative to the movable object.

[0037] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including a processor and a machine-readable storage medium. The machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is caused by the machine-executable instructions to implement any one of the above-mentioned methods for determining external parameters of a panoramic camera.

[0038] According to a fifth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any one of the above-mentioned methods for determining external parameters of a panoramic camera.

[0039] Advantages of the embodiments of the present application:

[0040] A method and device for determining the external parameters of a surround-view camera and a surround-view bird's-eye view acquisition system provided by an embodiment of the present application can map the same spatial points in the ground common view area to a first virtual image and a second virtual image respectively. Since the first virtual image and the second virtual image are respectively images acquired by a simulated first virtual camera and a second virtual camera, and the first virtual camera and the second virtual camera face the ground common view area, the ground common view area of the first surround-view camera and the second surround-view camera is also the ground common view area of the first virtual camera and the second virtual camera. Therefore, based on the projection points of the spatial points in the ground common view area on the first virtual image and the second virtual image, the external parameters of the first virtual camera and the second virtual camera can be determined. Also, since the position of the first virtual camera is the same as that of the first surround-view camera, and although the orientations of the first virtual camera and the first surround-view camera are different, the orientation of the first virtual camera relative to the first surround-view camera is known. Therefore, the external parameters of the first surround-view camera can be calculated based on the external parameters of the first virtual camera. Similarly to the first virtual camera and the first surround-view camera, the external parameters of the second surround-view camera can also be calculated based on the external parameters of the second virtual camera. That is, by determining the external parameters of the virtual camera and the second virtual camera, the external parameters of the first surround-view camera and the second surround-view camera can be realized. On the other hand, the projection points used to determine the external parameters in this application are the projection points in the first virtual image and the second virtual image. Also, since the external parameters of the first virtual camera are the relative external parameters relative to the first surround-view camera, and this relative external parameter does not change with the change of the external parameters of the first surround-view camera during the calibration process, the image coordinate system to which the first virtual image belongs is also fixed during the calibration process. There is no need to regenerate the virtual image during the calibration process, nor to determine new projection points, that is, there is no need to re-project the spatial points onto the first virtual image. Only the first projection point needs to be re-projected onto the second virtual image, effectively reducing the computational complexity required to select the projection points and improving the calibration efficiency.

[0041] Of course, implementing any product or method of the present application does not necessarily require achieving all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other embodiments can also be obtained based on these drawings.

[0043] Figure 1a It is a schematic diagram of the layout of surround-view cameras provided by an embodiment of the present application;

[0044] Figure 1b It is a schematic diagram of an accurately stitched surround-view bird's-eye view provided by the present application;

[0045] Figure 1c Schematic diagram of the panoramic bird's-eye view generated without accurate stitching provided by this application;

[0046] Figure 1d Schematic diagram of the ground co-visible area provided by this application;

[0047] Figure 1e Schematic diagram of the optical axis of the panoramic camera when the vehicle is not tilted provided by this application;

[0048] Figure 1f Schematic diagram of the optical axis of the panoramic camera when the vehicle is tilted provided by this application;

[0049] Figure 2 Schematic diagram of a process for determining the external parameters of the panoramic camera provided by this application;

[0050] Figure 3a Schematic diagram of the virtual camera provided by this application;

[0051] Figure 3b Schematic diagram of the virtual image provided by this application;

[0052] Figure 3c Schematic diagram of the original image provided by this application;

[0053] Figure 4 Schematic diagram of another process for determining the external parameters of the panoramic camera provided by this application;

[0054] Figure 5 Schematic diagram of yet another process for determining the external parameters of the panoramic camera provided by this application;

[0055] Figure 6 Schematic diagram of another process for determining the external parameters of the panoramic camera provided by this application;

[0056] Figure 7 Schematic diagram of yet another process for determining the external parameters of the panoramic camera provided by this application;

[0057] Figure 8 Schematic diagram of the parameter iteration process in the method for determining the external parameters of the panoramic camera provided by this application;

[0058] Figure 9 Schematic diagram of a structure of the device for determining the external parameters of the panoramic camera provided by this application;

[0059] Figure 10 Schematic diagram of a structure of the electronic device provided by this application. Detailed implementation manners

[0060] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0061] To more clearly illustrate the method for determining the extrinsic parameters of the surround-view camera provided by the present application, an exemplary description will be given below of a possible application scenario of the method for calibrating the extrinsic parameters of the surround-view camera provided by the present application. It can be understood that the following examples are only a possible application scenario of the method for determining the extrinsic parameters of the surround-view camera provided by the present application. In other possible embodiments, the method for determining the extrinsic parameters of the surround-view camera provided by the present application can also be applied to other possible embodiments, and the following examples do not impose any restrictions on this.

[0062] Vehicles with autonomous driving or assisted driving functions need to accurately identify traffic signs on the ground around the vehicle, such as zebra crossings, turning arrows, etc., in order to achieve autonomous driving or assisted driving functions. Therefore, as Figure 1a shown, surround-view cameras are respectively arranged in the front, left, right, and rear of the vehicle. During the driving process of the vehicle, each surround-view camera respectively collects image data, generates a bird's-eye view based on the image data collected by each surround-view camera, and stitches the generated bird's-eye views according to the extrinsic parameters of each camera to obtain a surround-view bird's-eye view as Figure 1b shown.

[0063] However, if the extrinsic parameters of each surround-view camera cannot be accurately determined, it will result in an inability to accurately stitch the bird's-eye views into a surround-view bird's-eye view. There may be a certain offset at the stitching position of each bird's-eye view, forming a surround-view bird's-eye view as Figure 1c shown. As Figure 1c can be seen, there is a break in the zebra crossing in the stitched surround-view bird's-eye view.

[0064] It can be understood that there is a certain overlap in the fields of view of adjacent surround-view cameras. In this article, this overlapping area is referred to as the ground common view area of the two surround-view cameras. As Figure 1a shown in the layout of the surround-view cameras, the ground common view area is as Figure 1d shown, where the area surrounded by the thick solid line is the ground common view area.

[0065] Spatial points in the ground co-view area will be captured by two adjacent surround-view cameras simultaneously. Therefore, pixel points corresponding to this spatial point exist in the data of the images collected by the two surround-view cameras. For convenience of description, the two adjacent surround-view cameras are respectively denoted as the first surround-view camera and the second surround-view camera, and this spatial point is denoted as point P. In the image data collected by the first surround-view camera, point P corresponds to pixel point x1, and in the image data collected by the second surround-view camera, point P corresponds to pixel point x2. Since the pixel value of a pixel point depends on the brightness of the corresponding spatial point, theoretically, the pixel value I1 of pixel point x1 is equal to the pixel value I2 of pixel point x2.

[0066] The pixel coordinates of pixel point x1 can be determined according to the extrinsic parameters of the first surround-view camera and the spatial coordinates of spatial point P. Similarly, the pixel coordinates of pixel point x2 can be determined according to the extrinsic parameters of the second surround-view camera and the spatial coordinates of spatial point P. When the extrinsic parameters of the first surround-view camera and the second surround-view camera are accurate, there is formula (1):

[0067] I1(f(P)) - I2(g(P)) = 0 … (1)

[0068] Where, I1(·) is the correspondence between pixel coordinates and pixel values in the image data collected by the first surround-view camera, I2(·) is the correspondence between pixel coordinates and pixel values in the image data collected by the second surround-view camera, f(·) is the coordinate conversion relationship between the spatial coordinate system and the image coordinate system of the first surround-view camera, g(·) is the coordinate conversion relationship between the spatial coordinate system and the image coordinate system of the second surround-view camera, and P is the spatial coordinate of point P.

[0069] However, if the extrinsic parameters of the first surround-view camera and the second surround-view camera are inaccurate, then there will be a certain difference between f(P) and pixel point x1. At the same time, there will also be a certain difference between g(P) and pixel point x2. Therefore, the right side of formula (1) is not strictly 0, but equal to a residual. Denote this residual as loss, then formula (1) is rewritten as formula (2):

[0070] I1(f(P)) - I2(g(P)) = loss … (2)

[0071] Formula (2) can be regarded as a function with the extrinsic parameters of the first surround camera and the second surround camera as independent variables and loss as the dependent variable. From formula (1), it can be seen that when the extrinsic parameters of the first surround camera and the second surround camera are the true extrinsic parameters, the loss is 0. Therefore, it can be considered that the closer the loss is to 0, the closer the extrinsic parameters of the first surround camera and the second surround camera are to the true extrinsic parameters. Therefore, the gradient descent direction of the loss can be determined, and the extrinsic parameters of the first surround camera and the second surround camera are adjusted along this gradient direction, so that the adjusted extrinsic parameters are closer to the true extrinsic parameters, that is, the correction of the extrinsic parameters is realized.

[0072] However, there are the following problems in the correction process of the extrinsic parameters:

[0073] Problem 1: This scheme simply constructs the loss based on the ground space points in the co-visible area. Therefore, it is equivalent to correcting the extrinsic parameters of the first surround camera and the second surround camera in the space coordinate system, that is, correcting the extrinsic parameters of the first surround camera and the second surround camera relative to the ground. However, in the actual working process, the surround camera is installed on the vehicle. Therefore, the images captured by the surround camera not only depend on the extrinsic parameters of the surround camera relative to the ground, but also depend on the pose of the vehicle, and the pose of the vehicle may be different according to different application scenarios.

[0074] Exemplarily, referring to Figure 1e and Figure 1f , Figure 1e and Figure 1f In the scenarios shown, the poses of the vehicle are different. Based on the above scheme, in Figure 1e the scenario shown, the extrinsic parameters of the first surround camera and the second surround camera are corrected so that the images captured by the first surround camera and the second surround camera can be accurately stitched. However, if the corrected pose is applied to Figure 1f the scenario shown, then as Figure 1f shown, there will be a stitching seam after the images captured by the first surround camera and the second surround camera are stitched. It can be seen that it is difficult to accurately correct the pose of the surround camera installed on the vehicle according to the above scheme.

[0075] Problem 2: The image has a certain continuity. Therefore, the pixel value of a pixel point in the image may be close to the pixel values of other pixel points around this pixel point, that is, the gradient of this pixel point is small. If the gradient at point f(P) in the image data captured by the first surround camera is small (similarly, the gradient at point g(P) will also be small), then even if there is a certain difference between f(P) and pixel point x1, and there is a certain difference between g(P) and pixel point x2, since the pixel values of pixel point x1 and its surrounding pixel points are similar, and the pixel values of pixel point x2 and its surrounding pixel points are similar, the loss is still small. As a result, it is wrongly considered that the extrinsic parameters are close to the true extrinsic parameters, that is, the corrected extrinsic parameters are still not accurate enough.

[0076] Therefore, to accurately correct the external parameters, it is necessary to select points with as large a gradient as possible for correction, such as at the contour line of the zebra crossing. However, during the process of correcting the external parameters, the external parameters of the surround-view camera will change with the correction. As the external parameters change, the image coordinate system will also change, resulting in the inability to continue using the previously selected pixel points. Therefore, it is necessary to continuously re-select the pixel points for correction. At the same time, as analyzed above, the pixel points for correction cannot be randomly selected, but rather pixel points with as large a gradient as possible need to be selected. Therefore, selecting pixel points also requires a certain amount of system resources. In this solution, since pixel points need to be repeatedly selected multiple times, it causes excessive consumption of system resources, thereby resulting in low correction efficiency.

[0077] Based on this, the present application provides a method for determining the external parameters of a surround-view camera, as Figure 2 shown, including:

[0078] S201, generating a first virtual image obtained by a first virtual camera collecting a ground common view area based on the image data collected by the first surround-view camera.

[0079] Among them, the first virtual camera has the same position as the first surround-view camera and faces the ground common view area. The ground common view area is the overlapping area between the fields of view of the first surround-view camera and the second surround-view camera. The first surround-view camera and the second surround-view camera are disposed on a movable object.

[0080] S202, generating a second virtual image obtained by a second virtual camera collecting a ground common view area based on the image data collected by the second surround-view camera.

[0081] Among them, the second virtual camera has the same position as the second surround-view camera and faces the ground common view area.

[0082] S203, determining a first projection point of a spatial point in the ground common view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial external parameters of the first surround-view camera and the second surround-view camera and the initial pose of the movable object.

[0083] S204, adjusting the initial external parameters of the first surround-view camera and the second surround-view camera according to a first difference between the pixel values of the first projection point and the second projection point to obtain the corrected external parameters of the first surround-view camera and the second surround-view camera relative to the movable object.

[0084] By selecting this embodiment, on the one hand, the same spatial point in the ground common view area is mapped to the first virtual image and the second virtual image respectively. Since the first virtual image and the second virtual image are images acquired by the simulated first virtual camera and the second virtual camera respectively, and the first virtual camera and the second virtual camera face the ground common view area, the ground common view area of ​​the first surround view camera and the second surround view camera is also the ground common view area of ​​the first virtual camera and the second virtual camera. Therefore, based on the projection points of the spatial points in the ground common view area on the first virtual image and the second virtual image, the external parameters of the first virtual camera and the second virtual camera can be determined. Since the first virtual camera and the first surround view camera have the same position, and although the first virtual camera and the first surround view camera have different orientations, the orientation of the first virtual camera relative to the first surround view camera is known. Therefore, the external parameters of the first surround view camera can be inferred according to the external parameters of the first virtual camera. Similarly, as with the first virtual camera and the first surround view camera, the external parameters of the second surround view camera can also be inferred according to the external parameters of the second virtual camera. That is, by determining the external parameters of the virtual camera and the second virtual camera, the external parameters of the first surround view camera and the second surround view camera can be determined.

[0085] On the other hand, since in the present application, not only the initial extrinsic parameters but also the initial posture of the movable object are considered when constructing the first difference, the first difference can not only reflect the difference in pixel values ​​between the first projection point and the second projection point caused by the initial extrinsic parameters, but also the difference in pixel values ​​between the first projection point and the second projection point caused by the posture of the movable object. Therefore, when adjusting the initial extrinsic parameters according to the first difference, the influence of the initial posture on the difference in pixel values ​​can be eliminated to a certain extent, so that the difference in pixel values ​​caused by the initial extrinsic parameters can be eliminated as much as possible by adjusting the initial extrinsic parameters. Therefore, the corrected extrinsic parameters obtained after adjustment are almost (or even completely) not affected by the posture of the movable object, that is, the corrected extrinsic parameters obtained after adjustment can be regarded as the external parameters of the surround camera relative to the movable object. Therefore, for movable objects in different postures, the corrected extrinsic parameters can be used to accurately perform image stitching, that is, the above-mentioned problem 1 can be solved.

[0086] On the other hand, the projection points used to determine the external parameters in the present application are projection points in the first virtual image and the second virtual image. Since the external parameters of the first virtual camera are relative external parameters with respect to the first surround-view camera, the relative external parameters will not change with the change of the external parameters of the first surround-view camera during the correction process. Therefore, the image coordinate system to which the first virtual image belongs is also fixed during the correction process. There is no need to regenerate the virtual image during the correction process, nor is there a need to determine new projection points. That is, there is no need to re-project the spatial points to the first virtual image, but only to re-project the first projection points to the second virtual image, which effectively reduces the amount of calculation required for selecting the projection points and improves the correction efficiency, thereby solving the above-mentioned problem 2.

[0087] The following will provide a detailed description of the aforementioned S201 - S204:

[0088] In S201, the virtual camera does not physically exist but is only a logically existing camera. The virtual image is an image obtained through simulation, which is the image that the virtual camera should capture assuming its real existence. The position of the first virtual camera being the same as that of the first surround camera means that the first virtual camera is assumed to be in the same position as the first surround camera.

[0089] The field of view of the first virtual camera should at least include the ground co - view area. And to make the first virtual image include as much of the complete ground co - view area as possible, the ground co - view area should be as close as possible to the center of the field of view of the first virtual camera. Therefore, the optical axis of the first virtual camera should be as close as possible to the center of the ground co - view area.

[0090] The assumed internal parameters of the first virtual camera are different from those of the first surround camera. In order to collect as much surrounding image information as possible, the first surround camera is often a camera with a large field of view angle, such as a fish - eye camera. For the first virtual camera, in order to reduce the image distortion of the first virtual image caused by the affine phenomenon, the field of view angle of the first virtual camera should be as small as possible while still covering the ground co - view area.

[0091] The relationship between the first virtual camera and the first surround camera can be as Figure 3a shown. It can be seen that the two cameras are in the same position. The first surround camera is a large - field - of - view camera with its optical axis facing directly forward (i.e., Figure 3a the lower part in ). The optical axis of the first virtual camera is obtained by rotating the optical axis of the first surround camera and is aligned with the ground co - view area, and the field of view angle of the first virtual camera is significantly smaller than that of the first surround camera.

[0092] The images captured are respectively as Figure 3b , Figure 3c shown. Among them, Figure 3b is the first virtual image, Figure 3c is the image data captured by the first surround camera (hereinafter referred to as the original image). As can be seen from Figure 3a , the ground co - view area is at the edge of the field of view of the first surround camera. Therefore, on the one hand, the affine phenomenon is serious, that is, the image distortion is serious, which makes it difficult to accurately determine the gradient based on the original image, that is, it is difficult to reasonably select the projection points for correction according to the image data captured by the first surround camera.

[0093] On the other hand, for some spatial points in the ground co-view area that exceed the 90° field of view boundary line, according to the principle of pinhole imaging, the size of the area where these spatial points are located will become infinite. Therefore, it is impossible to select the projection points corresponding to these spatial points for correction. As a result, the number of projection points that can be selected for correction is reduced, and it may not be possible to accurately perform the correction due to the inability to select a sufficient number of projection points.

[0094] In the first virtual image, since the ground co-view area is close to the center of the field of view in the first virtual camera, the aforementioned problem does not exist. That is, the gradient can be determined relatively accurately based on the first virtual image, thereby selecting the projection points. At the same time, the projection points corresponding to each spatial point in the ground co-view area can be selected to correct the initial extrinsic parameters, reducing the possibility of being unable to accurately perform the correction due to the inability to select a sufficient number of projection points.

[0095] The movable object in this application can be any device with the ability to move, including but not limited to vehicles, movable guiding robots, etc.

[0096] In S202, the relationship between the second omnidirectional camera and the second virtual camera is the same as that between the first omnidirectional camera and the first virtual camera described above. For the relevant description of the first omnidirectional camera and the first virtual camera in S201, reference can be made, and details will not be repeated here.

[0097] In S203, the first projection point is the projection point of the spatial point in the first virtual image, and the second projection point is the projection point of the first projection point in the second virtual image. It can be understood that if the initial extrinsic parameters and the initial pose are accurate enough, theoretically the second projection point is the projection point of the same spatial point in the second virtual image, that is, the second projection point is the projection point of the same spatial point in the second virtual image. Therefore, theoretically, the first difference in the pixel values of the first projection point and the second projection point should be 0. In a possible embodiment, spatial points can be selected, and the selected spatial points are projected onto the first virtual image to obtain the first projection point and projected onto the second virtual image to obtain the second projection point.

[0098] The initial pose in this application can be a pose determined according to any pose determination method, or a pose obtained by correcting the determined pose. This application does not impose any limitations on the pose correction method used for correcting the pose, and an exemplary description of how to correct the pose of the movable object will be given below, and details will not be repeated here.

[0099] It can be understood that, as described above, to improve the accuracy of the determined extrinsic parameters, the gradient at the projection point should be made as large as possible. However, it is difficult to determine whether the gradient at the projection point corresponding to a spatial point is large enough when selecting the spatial point. Therefore, in a possible embodiment, a first projection point with a large enough gradient can be selected in the first virtual image, and the first projection point is projected onto the second virtual image to obtain a second projection point. A large enough gradient in this text can mean that the gradient is greater than a preset gradient threshold, or it can mean the first N projection points with the largest gradients, where N is any positive integer.

[0100] In another possible embodiment, it can also be based on the image data collected by the first panoramic camera to generate a bird's-eye view of the ground co-view area, and determine the pixel points whose gradients meet the preset gradient condition in the bird's-eye view as the target pixel points. The target pixel points are projected onto the first virtual image to obtain a first projection point, and the first projection point is projected onto the second virtual image to obtain a second projection point.

[0101] Among them, the preset gradient condition is that the gradient is large enough. It can be understood that, compared with the first virtual image, the bird's-eye view can be regarded as a front view of the ground, which can effectively avoid the distortion of the ground image caused by the shooting angle. Therefore, based on the bird's-eye view, the gradients of each pixel point can be determined more accurately, so as to select the projection points with large enough gradients.

[0102] The initial extrinsic parameters can be the extrinsic parameters determined by any extrinsic parameter determination method. Exemplarily, the initial extrinsic parameters can be the calibrated extrinsic parameters determined in advance using the panoramic camera external acquisition method provided by this application, or the extrinsic parameters determined in advance using other extrinsic parameter determination methods other than the panoramic camera external acquisition method provided by this application. This application does not make any restrictions on this.

[0103] In S204, as described above, theoretically, the first difference between the first projection point and the second projection point should be 0. Therefore, the first difference can reflect the difference between the initial extrinsic parameters and the true extrinsic parameters. By adjusting the initial extrinsic parameters to make the first difference approach 0, the calibrated extrinsic parameters that are more accurate than the initial extrinsic parameters can be determined.

[0104] To more clearly illustrate the panoramic camera extrinsic parameter determination method provided by this application, it will be described below in conjunction with specific legends. Refer to Figure 1a , for the convenience of description, the four panoramic cameras are respectively called the left camera, the right camera, the front camera, and the rear camera. Among them, the left camera is the Figure 1a panoramic camera located on the left in Figure 1a , the right camera is the Figure 1a panoramic camera located on the right in

[0105] The process is asFigure 4 As shown in Figure 4 , first, based on the image data and initial extrinsic parameters collected by the left camera and the right camera, bird's-eye views corresponding to the left camera and the right camera are respectively generated. For the convenience of description, the bird's-eye view corresponding to the left camera is called the left bird's-eye view, and the bird's-eye view corresponding to the right camera is called the right bird's-eye view.

[0106] In the left bird's-eye view, pixel points with a large enough gradient in the ground common view area between the left camera and the rear camera are selected as the first target pixel points. In the right bird's-eye view, pixel points with a large enough gradient in the ground common view area between the right camera and the rear camera are selected as the second target pixel points.

[0107] Based on the initial extrinsic parameters, the first target pixel points are projected onto the left virtual image collected by the left virtual camera to obtain left projection points, where the position of the left virtual camera is the same as that of the left camera and faces the ground common view area between the left camera and the rear camera. And the second target pixel points are projected onto the right virtual image collected by the right virtual camera to obtain right projection points, where the position of the right virtual camera is the same as that of the right camera and faces the ground common view area between the right camera and the rear camera.

[0108] Based on the initial extrinsic parameters, the left projection points are projected onto the first rear virtual image collected by the first rear virtual camera to obtain first rear projection points, where the position of the rear virtual camera is the same as that of the rear camera and faces the ground common view area between the left camera and the rear camera. And the right projection points are projected onto the second rear virtual image collected by the second rear virtual camera to obtain second rear projection points, where the position of the second rear virtual camera is the same as that of the rear camera and faces the ground common view area between the right camera and the rear camera.

[0109] Based on the difference in pixel values between the left projection points and the first rear projection points, and the difference in pixel values between the right projection points and the second rear projection points, the initial extrinsic parameters of the left camera, the right camera, and the rear camera are adjusted to reduce the sum of these two differences, and the corrected extrinsic parameters of the left camera, the right camera, and the rear camera are obtained. The adjustment method can be as described in the aforementioned formulas (1) and (2), and the initial extrinsic parameters are adjusted by minimizing the residual loss. Therefore, this step can also be called optimizing the surround-view extrinsic parameters to minimize the residual.

[0110] In this example, the left camera and the right camera are equivalent to the aforementioned first surround-view cameras, the rear camera is equivalent to the aforementioned second surround-view camera, the left virtual camera and the right virtual camera are equivalent to the aforementioned first virtual camera, and the first rear virtual camera and the second rear virtual camera are equivalent to the aforementioned second virtual camera. The left projection points and the right projection points are equivalent to the aforementioned first projection points, and the first rear projection points and the second rear projection points are equivalent to the aforementioned second projection points. The relevant descriptions above can be referred to and will not be elaborated here.

[0111] It can be understood that the determined corrected pose may still not be accurate enough. In one possible embodiment, in order to further improve the accuracy of the corrected pose, the corrected pose can be corrected to obtain a more accurate new corrected pose.

[0112] Exemplarily, as Figure 5 shown, Figure 5 shown is another flowchart of the method for determining the extrinsic parameters of the surround cameras provided by this application, including:

[0113] S501, based on the image data collected by the first surround camera, generate a first virtual image obtained by a first virtual camera capturing the ground co-visible area.

[0114] This step is the same as the foregoing S201. For relevant descriptions, reference can be made to the foregoing S201 and will not be elaborated here.

[0115] S502, based on the image data collected by the second surround camera, generate a second virtual image obtained by a second virtual camera capturing the ground co-visible area.

[0116] This step is the same as the foregoing S202. For relevant descriptions, reference can be made to the foregoing S202 and will not be elaborated here.

[0117] S503, according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object, determine the first projection point of the spatial point in the ground co-visible area in the first virtual image and the second projection point of the first projection point in the second virtual image.

[0118] This step is the same as the foregoing S203. For relevant descriptions, reference can be made to the foregoing S203 and will not be elaborated here.

[0119] S504, according to the first difference in the pixel values of the first projection point and the second projection point, adjust the initial extrinsic parameters of the first surround camera and the second surround camera to obtain the corrected extrinsic parameters of the first surround camera and the second surround camera.

[0120] This step is the same as the foregoing S204. For relevant descriptions, reference can be made to the foregoing S204 and will not be elaborated here.

[0121] S505, according to the corrected extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object, determine the third projection point of the first projection point in the second virtual image.

[0122] As described above, the first projection point does not change. Therefore, this step is equivalent to re-projecting the first projection point onto the second virtual image according to the corrected extrinsic parameters to obtain the third projection point.

[0123] S506. Until the second difference meets the preset convergence condition, adjust the corrected external parameters according to the second difference to obtain new corrected external parameters, and then return to execute S505.

[0124] Among them, the second difference is the difference in pixel values between the first projection point and the third projection point. The preset convergence condition can be that the second difference is less than the preset difference threshold, or that the change in the second difference in this cycle compared to the second difference in the previous cycle is less than the preset change threshold, that is, the second difference has converged sufficiently.

[0125] It can be understood that if the second difference meets the preset convergence condition, it can be considered that the corrected external parameters are already close enough to the true external parameters, so there is no need to re-determine new corrected external parameters. On the contrary, if the second difference does not meet the preset convergence condition, it can be considered that there is still a certain gap between the corrected external parameters and the true external parameters, and it is necessary to further correct the corrected external parameters to obtain more accurate corrected external parameters.

[0126] Selecting this embodiment can continuously correct the corrected external parameters through an iterative method until the corrected external parameters are close enough to the true external parameters, thereby further improving the accuracy of the obtained corrected external parameters.

[0127] Still taking the application scenario of determining Figure 1a the external parameters of the left camera, right camera, and rear camera as an example, then Figure 5 the shown process can be as Figure 6 shown. The process before the first determination of the corrected external parameters is the same as Figure 4 that, and the relevant description in Figure 4 can be referred to, which will not be elaborated here.

[0128] After the first determination of the corrected external parameters, project the left projection point onto the first virtual rear image according to the new corrected external parameters to obtain a new first rear projection point, and project the right projection point onto the second virtual rear image to obtain a new second rear projection point. Based on the new first rear projection point and the new second rear projection point, determine whether the difference between the left projection point and the first rear projection point and the difference between the right projection point and the second rear projection point are small enough. If they are small enough, end the correction; if not, continue to adjust the corrected external parameters based on the new first rear projection point and the new second rear projection point.

[0129] Figure 4 、 Figure 6As shown, these are merely two possible flow diagrams of the method for determining the extrinsic parameters of the surround-view cameras provided by this application. In other possible embodiments, the method for determining the extrinsic parameters of the surround-view cameras provided by this application can be used to simultaneously determine the extrinsic parameters of more than three surround-view cameras. It can be understood that, as described above, since the method for determining the extrinsic parameters of the surround-view cameras provided by this application only needs to select projection points once for each image frame, the computational amount generated by selecting projection points can be effectively reduced. Therefore, when the hardware performance is limited, more extrinsic parameters of surround-view cameras can be determined simultaneously.

[0130] The pose of the movable object may change over time. Exemplarily, the initial pose of the movable object is as Figure 1e shown, and as the movable object moves, the pose gradually changes to Figure 1f shown. If this change is ignored, it may cause the bird's-eye views corresponding to two surround-view cameras to not be accurately stitched together, forming a surround-view bird's-eye view as Figure 1c shown.

[0131] Based on this, in a possible embodiment, as Figure 7 shown, it includes:

[0132] S701, based on the image data collected by the first surround-view camera, generate a first virtual image obtained by the first virtual camera capturing the ground co-visible area.

[0133] This step is the same as the aforementioned S201, and the relevant description of S201 can be referred to, and will not be elaborated here.

[0134] S702, based on the image data collected by the second surround-view camera, generate a second virtual image obtained by the second virtual camera capturing the ground co-visible area.

[0135] This step is the same as the aforementioned S202, and the relevant description of S202 can be referred to, and will not be elaborated here.

[0136] S703, according to the initial extrinsic parameters of the first surround-view camera and the second surround-view camera and the initial pose of the movable object, determine the first projection point of the spatial point in the ground co-visible area in the first virtual image and the second projection point of the first projection point in the second virtual image.

[0137] In this embodiment, the initial extrinsic parameters and the subsequent corrected extrinsic parameters both refer to the extrinsic parameters of the surround-view camera relative to the movable object. The pose of the movable object includes: the height of the movable object and the inclination angle of the movable object relative to the ground.

[0138] S704, adjusting the initial extrinsic parameters and the initial posture according to the first difference in pixel values ​​between the first projection point and the second projection point, to obtain a corrected posture of the movable object and corrected extrinsic parameters of the first surround-view camera and the second surround-view camera relative to the movable object in the corrected posture.

[0139] By selecting this embodiment, while correcting the initial external parameters, the posture of the movable object is used as the object of correction. By correcting the initial external parameters and the initial posture at the same time, the impact of the change of the posture of the movable object on the surround-view camera is taken into account, thereby reducing the possibility of not being able to accurately stitch together to obtain a surround-view bird's-eye view due to the change of the posture of the movable object.

[0140] It is understandable that, although the first difference and the second difference will be equal to 0 when the extrinsic parameter is equal to the true extrinsic parameter, it is actually difficult to determine the extrinsic parameter that makes the first difference and the second difference exactly equal to 0 due to various conditions, and it is only possible to find the extrinsic parameter that makes the first difference and the second difference as close to 0 as possible. However, due to various reasons, such as the aforementioned gradient is too small, scale drift phenomenon, etc., when the extrinsic parameter is far from the true extrinsic parameter, the first difference and the second difference may also approach 0, which leads to the inaccuracy of the determined correction parameters.

[0141] Based on this, in a possible embodiment, a priori constraints can be added to constrain the determined correction extrinsic parameters to be as close to the true extrinsic parameters as possible, reducing the possibility of mistakenly determining the wrong extrinsic parameters far away from the true extrinsic parameters as the correction extrinsic parameters. Exemplarily, in a possible embodiment, when adjusting the initial extrinsic parameters, in addition to the aforementioned first difference, it is also based on the degree of matching, where the degree of matching refers to: the degree of matching between the position of the first surround view camera and the position of the second surround view camera and the prior position condition.

[0142] The prior position condition may be a condition that is set based on experience or actual conditions and that the positions of the first surround-view camera and the second surround-view camera should satisfy. For example, if, based on the installation positions of the first surround-view camera and the second surround-view camera, it can be estimated in advance that the first surround-view camera is located 5 meters directly in front of the second surround-view camera, then the prior position condition may be: the position of the first surround-view camera is 5 meters directly in front of the position of the second surround-view camera.

[0143] If, on this basis, the initial position of the movable object is corrected at the same time, the correction process can be regarded as Figure 9 Parameter iteration diagram shown. Figure 8 In the example shown, the external parameters of the surround-view camera are composed of a rotation component and a translation component. The rotation component is used to represent the orientation of the surround-view camera, and the translation component is used to represent the position of the surround-view camera.

[0144] The figure is marked with "RBC xThe circular shape marked with " " represents the rotation component, with the subscript BC F The front camera, with the subscript BC L Represents the left camera, with the subscript BC R Represents the right camera, with the subscript BC B Represents the rear camera. Exemplarily, the circular shape marked with "RBC F " represents the rotation component in the extrinsic parameters of the front camera. The circular shape marked with "TBC x " represents the translation component. The meaning of each subscript is the same as that of the rotation component. Exemplarily, the circular shape marked with "RBC F " represents the translation component in the extrinsic parameters of the front camera. The extrinsic parameters in this example refer to the extrinsic parameters of the camera coordinate system relative to the vehicle body coordinate system.

[0145] In the figure, the circular shape marked with "Tilt&Height" represents the pose of the movable object. The star represents the residual, and the square with a thick frame represents the prior condition of the connected parameter. Among them, the white square with a thick frame represents the prior condition of the connected translation component. Exemplarily, the white square with a thick frame connected to the circular shape marked with "RBC F " in the figure represents the prior condition of the translation component of the front camera. The grey square with a thick frame represents the prior condition of the tilt angle. The connecting line represents the constraint relationship. It can be Figure 8 seen that the translation component is constrained by the prior position condition, and at the same time, the pose of the movable object is constrained by the prior tilt angle condition. Each parameter is constrained by minimizing the residual.

[0146] Corresponding to the foregoing method for determining the extrinsic parameters of a surround-view camera, the present application correspondingly provides a surround-view bird's-eye view acquisition system, which includes a first surround-view camera, a second surround-view camera, and a processor. There is an overlapping ground co-view area between the field of view of the first surround-view camera and the field of view of the second surround-view camera;

[0147] The first surround-view camera and the second surround-view camera are arranged on the movable object, and the first surround-view camera and the second surround-view camera are used to collect image data at different orientations of the movable object;

[0148] A processor is configured to generate a first virtual image of a ground common view area captured by a first virtual camera based on image data collected by a first surround camera; generate a second virtual image of the ground common view area captured by a second virtual camera based on image data collected by the second surround camera; determine a first projection point of a spatial point in the ground common view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object, and adjust the initial extrinsic parameters according to a first difference between the pixel values of the first projection point and the second projection point to obtain corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object; wherein, the first virtual camera has the same position as the first surround camera and faces the ground common view area, and the second virtual camera has the same position as the second surround camera and faces the ground common view area;

[0149] The processor is further configured to generate bird's-eye views at different orientations of the movable object based on correction parameters and the corrected extrinsic parameters of the first surround camera and the second surround camera; and stitch the bird's-eye views to obtain a surround bird's-eye view of the movable object.

[0150] Corresponding to the foregoing method for determining the extrinsic parameters of a surround camera, the present application correspondingly provides a device for determining the extrinsic parameters of a surround camera, as Figure 9 shown Figure 9 FIG. is a schematic structural diagram of a device for determining the extrinsic parameters of a surround camera provided by the present application, including:

[0151] A first generation module 901 is configured to generate a first virtual image of a ground common view area captured by a first virtual camera based on image data collected by a first surround camera, wherein the first virtual camera has the same position as the first surround camera and faces the ground common view area, the ground common view area is an overlapping area between the field of view of the first surround camera and the field of view of the second surround camera, and the first surround camera and the second surround camera are disposed on a movable object;

[0152] A second generation module 902 is configured to generate a second virtual image of the ground common view area captured by a second virtual camera based on image data collected by the second surround camera, wherein the second virtual camera has the same position as the second surround camera and faces the ground common view area;

[0153] A determination module 903 is configured to determine a first projection point of a spatial point in the ground common view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object;

[0154] An adjustment module 904, configured to adjust the initial external parameters according to a first difference between the pixel values of the first projection point and the second projection point, so as to obtain corrected external parameters of the first surround camera and the second surround camera relative to the movable object.

[0155] In a possible embodiment, the apparatus further includes:

[0156] Generate a bird's-eye view of the ground co-view area based on the image data collected by the first surround camera;

[0157] Determine, in the bird's-eye view, a pixel point whose gradient satisfies a preset gradient condition as a target pixel point, where the preset gradient condition includes: the gradient is greater than a preset gradient threshold;

[0158] Determining a first projection point of a spatial point in the ground co-view area in the first virtual image and a second projection point of the first projection point in the second virtual image includes:

[0159] Project the target pixel point onto the first virtual image to obtain a first projection point; and project the first projection point onto the second virtual image to obtain a second projection point.

[0160] In a possible embodiment, the apparatus further includes:

[0161] Adjusting the initial external parameters according to a first difference between the pixel values of the first projection point and the second projection point to obtain corrected external parameters of the first surround camera and the second surround camera relative to the movable object includes:

[0162] Adjust the initial external parameters and the initial pose according to the first difference between the pixel values of the first projection point and the second projection point, so as to obtain a corrected pose of the movable object, and corrected external parameters of the first surround camera and the second surround camera relative to the movable object in the corrected pose.

[0163] In a possible embodiment, the apparatus further includes:

[0164] Determine a third projection point of the first projection point in the second virtual image according to the corrected external parameters and the corrected pose;

[0165] Until a second difference between the pixel values of the first projection point and the third projection point satisfies a preset convergence condition, adjust the corrected external parameters and the corrected pose according to the second difference to obtain new corrected external parameters and a new corrected pose, and return to execute the step of determining a third projection point of the first projection point in the second virtual image according to the corrected external parameters and the corrected pose.

[0166] In a possible embodiment, the apparatus further includes:

[0167] Determine the matching degree between the position of the first surround camera and the position of the second surround camera and the prior position condition respectively according to the initial extrinsic parameters of the first surround camera and the second surround camera;

[0168] Adjust the initial extrinsic parameters according to the first difference of the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object, including:

[0169] Adjust the initial extrinsic parameters according to the first difference of the pixel values of the first projection point and the second projection point and the matching degree to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

[0170] In a possible embodiment, the device further includes:

[0171] Adjust the initial extrinsic parameters according to the first difference of the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object, including:

[0172] Determine the gradient descent direction of the loss function with the initial extrinsic parameters as the independent variable and the first difference as the dependent variable;

[0173] Adjust the initial extrinsic parameters along the gradient descent direction to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

[0174] An embodiment of the present application also provides an electronic device, as Figure 10 shown, including:

[0175] A memory 101 for storing a computer program;

[0176] A processor 102, when executing the program stored on the memory 101, implements the following steps:

[0177] Generate a first virtual image obtained by a first virtual camera collecting a ground co-view area based on the image data collected by the first surround camera, wherein the first virtual camera has the same position as the first surround camera and faces the ground co-view area, the ground co-view area is the overlapping area between the field of view of the first surround camera and the field of view of the second surround camera, and the first surround camera and the second surround camera are arranged on a movable object;

[0178] Generate a second virtual image obtained by a second virtual camera collecting the ground co-view area based on the image data collected by the second surround camera, wherein the second virtual camera has the same position as the second surround camera and faces the ground co-view area;

[0179] Determine a first projection point of a spatial point within the ground co-view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object;

[0180] Adjust the initial extrinsic parameters according to a first difference between the pixel values of the first projection point and the second projection point to obtain corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

[0181] Moreover, the above-mentioned electronic device may further include a communication bus and / or a communication interface, and the processor 102, the communication interface, and the memory 101 complete communication with each other through the communication bus.

[0182] The communication bus mentioned in the above-mentioned electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used in the figure to represent it, but it does not mean that there is only one bus or one type of bus.

[0183] The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0184] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0185] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0186] In another embodiment provided by the present application, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of any of the above methods for determining the extrinsic parameters of the surround-view camera are implemented.

[0187] In another embodiment provided by the present application, a computer program product including instructions is further provided. When it runs on a computer, the computer is caused to execute any of the methods for determining the extrinsic parameters of the surround-view camera in the above embodiments.

[0188] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a solid-state disk (SSD), etc.

[0189] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0190] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of devices, systems, electronic devices, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0191] The above are only the preferred embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application are all included in the protection scope of this application.

Claims

1. A method for determining the extrinsic parameters of a surround-view camera, characterized in that, The method includes: Generating a first virtual image obtained by a first virtual camera capturing a ground common view area based on image data collected by a first surround camera, where the first virtual camera has the same position as the first surround camera and faces the ground common view area, the ground common view area is an overlapping area between the field of view of the first surround camera and the field of view of a second surround camera, and the first surround camera and the second surround camera are disposed on a movable object; Generating a second virtual image obtained by a second virtual camera capturing the ground common view area based on image data collected by the second surround camera, where the second virtual camera has the same position as the second surround camera and faces the ground common view area; Determining a first projection point of a spatial point in the ground common view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to initial extrinsic parameters of the first surround camera and the second surround camera and an initial pose of the movable object; Adjusting the initial extrinsic parameters according to a first difference in pixel values of the first projection point and the second projection point to obtain corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

2. The method according to claim 1, wherein The method further includes: Generating a bird's-eye view of the ground common view area based on image data collected by the first surround camera; Determining, in the bird's-eye view, pixel points whose gradients satisfy a preset gradient condition as target pixel points, where the preset gradient condition includes: the gradient is greater than a preset gradient threshold; The determining the first projection point of the spatial point in the ground common view area in the first virtual image and the second projection point of the first projection point in the second virtual image includes: Projecting the target pixel point onto the first virtual image to obtain a first projection point; and projecting the first projection point onto the second virtual image to obtain a second projection point.

3. The method according to claim 1, wherein The adjusting the initial extrinsic parameters according to a first difference in pixel values of the first projection point and the second projection point to obtain corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object includes: Adjusting the initial extrinsic parameters and the initial pose according to a first difference in pixel values of the first projection point and the second projection point to obtain a corrected pose of the movable object and corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object in the corrected pose.

4. The method according to claim 3, wherein The method further includes: Determining a third projection point of the first projection point in the second virtual image according to the corrected extrinsic parameters and the corrected pose; Until a second difference in pixel values of the first projection point and the third projection point satisfies a preset convergence condition, adjusting the corrected extrinsic parameters and the corrected pose according to the second difference to obtain new corrected extrinsic parameters and a new corrected pose, and returning to execute the step of determining the third projection point of the first projection point in the second virtual image according to the corrected extrinsic parameters and the corrected pose.

5. The method according to claim 1, wherein The method further includes: According to the initial extrinsic parameters of the first surround camera and the second surround camera, respectively determine the matching degrees between the positions of the first surround camera and the second surround camera and the prior position conditions; The adjusting the initial extrinsic parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object includes: Adjust the initial extrinsic parameters according to the first difference between the pixel values of the first projection point and the second projection point and the matching degree to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

6. The method according to claim 1, characterized in that, The adjusting the initial extrinsic parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object includes: Determine the gradient descent direction of the loss function with the initial extrinsic parameters as the independent variable and the first difference as the dependent variable; Adjust the initial extrinsic parameters along the gradient descent direction to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

7. A surround view bird's-eye view acquisition system, characterized in that The system includes a first surround camera, a second surround camera, and a processor. There is an overlapping ground co-view area between the field of view of the first surround camera and the field of view of the second surround camera; The first surround camera and the second surround camera are disposed on the movable object, and the first surround camera and the second surround camera are used to collect image data at different orientations of the movable object; The processor is configured to generate a first virtual image obtained by a first virtual camera collecting the ground co-view area based on the image data collected by the first surround camera; generate a second virtual image obtained by a second virtual camera collecting the ground co-view area based on the image data collected by the second surround camera; determine a first projection point of a spatial point in the ground co-view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to the initial extrinsic parameters of the first surround camera and the second surround camera and the initial pose of the movable object, and adjust the initial extrinsic parameters according to the first difference between the pixel values of the first projection point and the second projection point to obtain the calibrated extrinsic parameters of the first surround camera and the second surround camera relative to the movable object; wherein, the first virtual camera has the same position as the first surround camera and is oriented towards the ground co-view area, and the second virtual camera has the same position as the second surround camera and is oriented towards the ground co-view area; The processor is further configured to generate an overhead view of different orientations of the movable object based on the calibration parameters and the calibrated extrinsic parameters of the first surround camera and the second surround camera; splice the overhead views to obtain an omnidirectional overhead view of the movable object.

8. An external parameter determination device for a surround camera, characterized in that The device includes: A first generation module, configured to generate a first virtual image obtained by a first virtual camera capturing a ground common view area based on image data collected by a first surround camera, wherein the first virtual camera has the same position as the first surround camera and faces the ground common view area, the ground common view area being an overlapping area between the field of view of the first surround camera and the field of view of a second surround camera, and the first surround camera and the second surround camera are disposed on a movable object; A second generation module, configured to generate a second virtual image obtained by a second virtual camera capturing the ground common view area based on image data collected by the second surround camera, wherein the second virtual camera has the same position as the second surround camera and faces the ground common view area; A determination module, configured to determine a first projection point of a spatial point in the ground common view area in the first virtual image and a second projection point of the first projection point in the second virtual image according to initial extrinsic parameters of the first surround camera and the second surround camera and an initial pose of the movable object; An adjustment module, configured to adjust the initial extrinsic parameters according to a first difference in pixel values of the first projection point and the second projection point to obtain corrected extrinsic parameters of the first surround camera and the second surround camera relative to the movable object.

9. An electronic device, characterized in that, Comprising a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions capable of being executed by the processor, the processor being urged by the machine-executable instructions to: implement the method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1-6 is implemented.

Images