Method and apparatus for generating an aerial view of a vehicle

Abstract

This application provides a method for generating a vehicle bird's-eye view image. The method includes: acquiring a first visual image from each of a plurality of visual sensors on the vehicle; converting the first visual image of each visual sensor into a bird's-eye view image based on the initial calibration parameters of each visual sensor to obtain a plurality of first bird's-eye view images; extracting and matching feature points in the overlapping areas of two adjacent first bird's-eye view images to obtain a plurality of corresponding bird's-eye view matching point pairs; back-mapping each bird's-eye view matching point pair back into the corresponding first visual image to obtain a plurality of visual matching point pairs; correcting the initial calibration parameters of each visual sensor based on the plurality of visual matching point pairs to obtain target calibration parameters; and mapping a second visual image of the visual sensor into a second bird's-eye view image based on the target calibration parameters and stitching them together to form a target bird's-eye view image, thereby improving the accuracy of the bird's-eye view image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a method and apparatus for generating a bird's-eye view of a vehicle. Background Technology

[0002] With the increasing intelligence of vehicles, surround view monitors (AVMs) have become an important feature of driver assistance systems. AVMs generate panoramic bird's-eye views to eliminate blind spots. However, in actual vehicle use, factors such as bumps, collisions, and maintenance (e.g., lens replacement, suspension adjustment) can cause slow or drastic changes to the system. Since AVMs use fixed calibration parameters to generate bird's-eye views, they cannot perceive these changes. This results in severe ghosting, misalignment, and geometric distortion in the generated images, leading to significant discrepancies between the generated bird's-eye view and the actual vehicle scene, making the generated bird's-eye view inaccurate. Summary of the Invention

[0003] This application provides a method and apparatus for generating a bird's-eye view of a vehicle, which at least solves the technical problem in the related art where the use of fixed calibration parameters during image generation results in a large error between the generated bird's-eye view and the actual scene of the vehicle, making the generated bird's-eye view inaccurate.

[0004] According to one aspect of the embodiments of this application, a method for generating a vehicle bird's-eye view is provided, comprising: acquiring a first visual image of each of a plurality of visual sensors on the vehicle, and converting the first visual image of each visual sensor into a bird's-eye view image based on initial calibration parameters of each visual sensor, thereby obtaining a plurality of first bird's-eye view images; each pair of adjacent visual sensors in the plurality of visual sensors has an overlapping region in the bird's-eye view; extracting feature points from the overlapping region of each first bird's-eye view image and adjacent first bird's-eye view images in the plurality of first bird's-eye view images, and matching the feature points of each pair of adjacent bird's-eye view images in the plurality of first bird's-eye view images to obtain a plurality of bird's-eye view images corresponding to each pair of visual sensors. The process involves: matching point pairs; each pair of bird's-eye view images corresponding to a pair of visual sensors; mapping each bird's-eye view matching point pair corresponding to each pair of visual sensors back to the first visual image of each pair of visual sensors, obtaining multiple visual matching point pairs corresponding to each pair of visual sensors; and correcting the initial calibration parameters of each visual sensor based on the multiple visual matching point pairs corresponding to each pair of visual sensors, obtaining the target calibration parameters of each visual sensor; acquiring the second visual image of each visual sensor; mapping the second visual image of each visual sensor to a bird's-eye view image based on the target calibration parameters of each visual sensor, obtaining multiple second bird's-eye view images, and stitching the multiple second bird's-eye view images together to form the target bird's-eye view image.

[0005] According to another aspect of the embodiments of this application, a vehicle bird's-eye view generation apparatus is also provided, comprising: a conversion unit, configured to acquire a first visual image of each of a plurality of visual sensors on the vehicle, and convert the first visual image of each visual sensor into a bird's-eye view image based on the initial calibration parameters of each visual sensor, thereby obtaining a plurality of first bird's-eye view images; wherein each pair of adjacent visual sensors in the plurality of visual sensors has an overlapping region in the bird's-eye view; and a matching unit, configured to extract feature points from the overlapping region of each first bird's-eye view image and adjacent first bird's-eye view images in the plurality of first bird's-eye view images, and match the feature points of each pair of adjacent bird's-eye view images in the plurality of first bird's-eye view images, thereby obtaining a plurality of bird's-eye view images corresponding to each pair of visual sensors. The system comprises: a matching point pair; each pair of bird's-eye view images corresponding to a pair of visual sensors; a correction unit, configured to back-map each bird's-eye view matching point pair corresponding to each pair of visual sensors back to the first visual image of each pair of visual sensors, thereby obtaining multiple visual matching point pairs corresponding to each pair of visual sensors, and correcting the initial calibration parameters of each visual sensor based on the multiple visual matching point pairs corresponding to each pair of visual sensors, thereby obtaining the target calibration parameters of each visual sensor; and an execution unit, configured to acquire the second visual image of each visual sensor, map the second visual image of each visual sensor to a bird's-eye view image based on the target calibration parameters of each visual sensor, thereby obtaining multiple second bird's-eye view images, and stitching the multiple second bird's-eye view images together to form a target bird's-eye view image.

[0006] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed by a processor.

[0007] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in any of the method embodiments described above.

[0008] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to perform the steps of any of the above method embodiments through the computer program.

[0009] This application acquires the first visual image of each of multiple visual sensors on a vehicle. Based on the initial calibration parameters of each visual sensor, the first visual image of each sensor is converted into a bird's-eye view image, resulting in multiple first bird's-eye view images. This causes adjacent visual sensors to form overlapping regions under the bird's-eye view. Feature points are then extracted and matched from the overlapping regions of two adjacent first bird's-eye view images to obtain multiple corresponding bird's-eye view matching point pairs. Each bird's-eye view matching point pair is then back-mapped back to the corresponding first visual image, resulting in multiple visual matching point pairs. Finally, based on these multiple visual matching point pairs, the initial calibration parameters of each visual sensor are corrected to obtain target calibration parameters. By utilizing the consistency constraints of multiple pairs of visual matching points in the overlapping region under multiple viewpoints, a geometric relationship is established in the original image space through an inverse mapping mechanism. The calibration parameters are directly optimized to eliminate reprojection errors, thereby correcting the deviation caused by the initial calibration. Finally, based on the corrected target calibration parameters, the second visual image of the visual sensor is mapped to a second bird's-eye view image and stitched together to form the target bird's-eye view image. This enables the generated bird's-eye view to accurately reflect the real geometric relationship of the vehicle's surrounding environment, improving the accuracy of the bird's-eye view. This can solve the technical problem in related technologies where the use of fixed calibration parameters during image generation leads to a large error between the generated bird's-eye view and the actual scene of the vehicle, resulting in inaccurate bird's-eye view images. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating an application scenario of a method for generating a vehicle bird's-eye view according to an embodiment of this application;

[0011] Figure 2 This is a flowchart illustrating an optional method for generating a vehicle bird's-eye view according to an embodiment of this application;

[0012] Figure 3 It is a bird's-eye view image obtained by two adjacent visual sensors according to an embodiment of this application;

[0013] Figure 4 This is a schematic diagram of an optional BEV image matching result according to an embodiment of this application;

[0014] Figure 5 The image is obtained after an optional consistency check according to an embodiment of this application;

[0015] Figure 6 This is a schematic diagram of an optional pre-optimized bird's-eye view stitching effect according to an embodiment of this application;

[0016] Figure 7 This is a schematic diagram of an optional optimized bird's-eye view stitching effect according to an embodiment of this application;

[0017] Figure 8 This is a flowchart illustrating the generation of an optional bird's-eye view according to an embodiment of this application;

[0018] Figure 9 This is a structural block diagram of an optional vehicle bird's-eye view generation device according to an embodiment of this application;

[0019] Figure 10 This is a computer system architecture block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] According to one aspect of the embodiments of this application, a method for generating a vehicle bird's-eye view is provided. Optionally, in this embodiment, the above-described method for generating a vehicle bird's-eye view can be applied, but is not limited to, to applications such as... Figure 1 The hardware environment shown includes vehicle 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide data storage services for vehicle 102. A database can be set up on server 104 or independently of server 104.

[0023] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wireless Fidelity (WIFI), Bluetooth. Server 104 may be, but is not limited to, a cloud server, server cluster, or other server type.

[0024] The method for generating a vehicle bird's-eye view in this application embodiment can be executed by server 104, by vehicle 102, or by both server 104 and vehicle 102.

[0025] Taking the vehicle bird's-eye view generation method in this embodiment as an example, which is executed by vehicle 102, Figure 2 This is a schematic flowchart of an optional method for generating a vehicle bird's-eye view according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0026] Step S202: Acquire the first visual image of each of the multiple visual sensors on the vehicle, and convert the first visual image of each visual sensor into a bird's-eye view image based on the initial calibration parameters of each visual sensor to obtain multiple first bird's-eye view images; each pair of adjacent visual sensors in the multiple visual sensors has an overlapping area in the bird's-eye view.

[0027] Step S204: Extract feature points from the overlapping area of ​​each first bird's-eye view image and its adjacent first bird's-eye view images in the multiple first bird's-eye view images, and match the feature points of each pair of adjacent bird's-eye view images in the multiple first bird's-eye view images to obtain multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors; each pair of bird's-eye view images corresponds to a pair of visual sensors.

[0028] Step S206: Map each bird's-eye view matching point pair corresponding to each pair of visual sensors back to the first visual image of each pair of visual sensors to obtain multiple visual matching point pairs corresponding to each pair of visual sensors. Based on the multiple visual matching point pairs corresponding to each pair of visual sensors, correct the initial calibration parameters of each visual sensor to obtain the target calibration parameters of each visual sensor.

[0029] Step S208: Obtain the second visual image of each visual sensor, map the second visual image of each visual sensor to a bird's-eye view image based on the target calibration parameters of each visual sensor, obtain multiple second bird's-eye view images, and stitch the multiple second bird's-eye view images into a target bird's-eye view image.

[0030] The method for generating a vehicle bird's-eye view in this embodiment can be applied to the field of image processing, specifically to scenarios involving the generation of such views. Optionally, the bird's-eye view can be generated when the vehicle is in reverse, allowing the driver to better understand the vehicle's surroundings and improving safety; or the bird's-eye view can be generated while the vehicle is in motion, allowing the driver to better understand the surroundings of the moving vehicle and improving safety.

[0031] A surround-view camera system generates a bird's-eye view panoramic image to eliminate blind spots. However, in the actual use of a vehicle, the system may undergo slow or drastic changes due to factors such as bumps, collisions, and maintenance (such as replacing lenses or adjusting suspension). Since the surround-view camera system generates the bird's-eye view image using fixed calibration parameters, it cannot perceive these changes, resulting in severe ghosting, misalignment, and geometric distortion in the generated image.

[0032] With the increasing intelligence of vehicles, the Around View Monitor (AVM) system has become an important feature of driver assistance. The AVM system collects images through multiple wide-angle fisheye lenses installed around the vehicle. After image correction, perspective transformation, and image stitching, it generates a seamless, bird's-eye view panoramic image of the vehicle's surrounding environment on the in-vehicle display screen. This system greatly assists the driver's visual perception in low-speed scenarios such as parking and navigating narrow roads, effectively reducing blind spots and improving driving safety.

[0033] However, the imaging quality of a surround-view system is highly dependent on the accuracy of lens calibration. Accurate calibration parameters are a prerequisite for precise image correction and seamless stitching. In related technologies, surround-view calibration methods typically rely on precise calibration targets (such as checkerboard or dot grids) and are performed under strictly controlled site conditions (such as level ground and specific lighting). This calibration process is usually completed by professionals before the vehicle leaves the factory or during maintenance at a 4S dealership, and is considered a form of "offline calibration."

[0034] However, the offline calibration methods in the above-mentioned related technologies have obvious technical limitations and pain points: (1) The calibration conditions are harsh and the cost is high: they rely on high-precision, large-size physical calibration objects and have strict requirements on the placement position and site flatness, which increases the manufacturing and maintenance costs. (2) They cannot adapt to parameter changes: during actual use of the vehicle, due to factors such as bumps, collisions, and maintenance (such as replacing lenses and adjusting suspension), the intrinsic parameters (such as focal length and principal point) and extrinsic parameters (installation position and angle) of the lens may change slowly or drastically. The offline calibration of the above-mentioned related technologies cannot perceive this change in real time, which leads to the failure of calibration parameters, which in turn causes serious ghosting, misalignment and geometric distortion in the surround view image, resulting in a large error between the generated bird's-eye view and the actual vehicle scene, making the generated bird's-eye view inaccurate and seriously affecting the user experience and safety. (3) Poor user experience: when the calibration fails, the user must return to a professional place for recalibration, which is cumbersome and time-consuming.

[0035] To overcome the above problems, various online self-calibration methods have been proposed in related technologies. These methods aim to automatically estimate and optimize lens parameters using natural scene images during vehicle driving without relying on specific calibration objects. The mainstream methods can be roughly divided into two categories: lane line-based methods and texture point feature loss-based methods. Among them, lane line-based methods rely on the detection and fitting of geometric structures such as lane lines in the image. They optimize calibration parameters by maximizing the geometric consistency of lane lines in the panoramic image under the bird's-eye view. However, they have the following limitations: (1) Strong feature dependence: They can only work effectively in road environments where lane lines are clear and completely visible. They will fail in parking lots, narrow streets, or areas without lane markings. (2) Poor robustness: Lane line detection is easily affected by lighting, shadows, stains, wear, etc., leading to false detections or false negatives, which causes the calibration optimization to fall into local optima or even divergence. (3) Limited applicability: It is not capable of handling cases where the features of the multi-lens stitching boundary area (the junction of the front and rear sides of the vehicle) are sparse and the angles are inconsistent. The texture point-based method optimizes the lens extrinsic parameters by extracting texture points in the overlapping area of ​​multiple lenses and calculating their photometric error, but it has the following limitations: (1) Texture points will be randomly selected within the maximum contour of the ground points. The number of points is large and the amount of calculation is large. Therefore, only a single frame image can be used to obtain the result. Otherwise, it cannot meet the real-time requirements. (2) Texture points may be taken from non-ground points. The back projection of non-ground points does not meet the assumption and will cause the loss function to fail to converge, resulting in optimization failure. (3) The photometric value of the texture points is greatly affected by the lens. The photometric difference between the two lenses is large and cannot converge. In addition, the photometric loss is greatly affected by the external lighting and is easy to introduce error factors.

[0036] To at least partially solve the aforementioned technical problems, this embodiment does not rely on specific geometric features such as lane lines, but also requires rich texture point features. It acquires the first visual image from each of the multiple visual sensors on the vehicle, and based on the initial calibration parameters of each visual sensor, converts the first visual image of each visual sensor into a bird's-eye view image, resulting in multiple first bird's-eye view images. This allows adjacent visual sensors to form overlapping areas from the bird's-eye view perspective. Feature points are then extracted and matched from the overlapping areas of adjacent first bird's-eye view images to obtain multiple corresponding bird's-eye view matching point pairs. Each bird's-eye view matching point pair is then back-mapped back to its corresponding first visual sensor. In the image, multiple visual matching point pairs are obtained. Then, based on the multiple visual matching point pairs, the initial calibration parameters of each visual sensor are corrected to obtain the target calibration parameters. By utilizing the consistency constraints of multiple pairs of visual matching points in the overlapping area under multiple views, a geometric relationship can be established in the original image space through an inverse mapping mechanism. The calibration parameters are directly optimized to eliminate reprojection errors, thereby correcting the deviation caused by the initial calibration. Finally, based on the corrected target calibration parameters, the second visual image of the visual sensor is mapped to a second bird's-eye view image and stitched together to form the target bird's-eye view image. This allows the generated bird's-eye view to accurately reflect the real geometric relationship of the environment around the vehicle, improving the accuracy of the bird's-eye view.

[0037] In this embodiment, each of the multiple vision sensors refers to a component used to acquire optical images of the environment surrounding the vehicle. Optionally, the vision sensors are mounted on the exterior of the vehicle and can be wide-angle cameras or fisheye cameras.

[0038] The first visual image of each visual sensor refers to the original image initially acquired by each visual sensor. Optionally, the first visual image of each visual sensor may be a wide-angle image with fisheye distortion.

[0039] The initial calibration parameters for each vision sensor refer to the calibration parameters of each vision sensor itself. Optionally, the initial calibration parameters for each vision sensor may include intrinsic parameters (referring to the internal optical characteristic parameters of the vision sensor, such as focal length, principal point coordinates, and distortion coefficients) and extrinsic parameters (referring to the parameters of the position and orientation of the vision sensor in the vehicle coordinate system, such as rotation matrix and translation vector).

[0040] A bird's-eye view (BEV) image is a top-down view image generated by projecting a three-dimensional spatial scene onto a two-dimensional virtual plane parallel to the ground (i.e., the bird's-eye view plane or BEV plane). Optionally, in this bird's-eye view image, the surrounding scenery of the vehicle is presented from a top-down perspective to facilitate the display of the environment around the vehicle.

[0041] In some embodiments, raw images (i.e., first visual images) captured at the current moment by multiple visual sensors around the vehicle are acquired. Then, using the initial calibration parameters of each visual sensor, the first visual images of each visual sensor are projected onto a virtual plane above the vehicle through perspective transformation to generate an image from a top-down perspective, i.e., a first bird's-eye view image. Since the field of view angles of the visual sensors installed around the vehicle overlap, the first bird's-eye view images generated by two adjacent visual sensors (such as the left front visual sensor and the front right visual sensor) will have a shared area in space, i.e., an overlapping area.

[0042] It should be noted that, in this embodiment, feature point extraction is performed on the overlapping area of ​​each first bird's-eye view image and its adjacent first bird's-eye view images. This allows the pixels in the overlapping area to correspond to the same three-dimensional physical point in the world coordinate system. Furthermore, by extracting features in the overlapping area, the range of feature points is significantly reduced, thereby lowering the mismatch rate of subsequent matching point pairs.

[0043] The first bird's-eye view image refers to the bird's-eye view image obtained by converting the first visual image using the initial calibration parameters of each visual sensor.

[0044] Each pair of vision sensors refers to two adjacent vision sensors among multiple vision sensors, and the overlapping area refers to the spatial range jointly covered by each adjacent pair of vision sensors in the bird's-eye view. The bird's-eye view is the spatial perspective from above the vehicle, looking vertically downwards at the environment surrounding the vehicle. Optionally, in the bird's-eye view, the ground plane is unfolded into a two-dimensional plane, and height information is compressed.

[0045] Optionally, based on the initial calibration parameters of each visual sensor, the first visual image of each visual sensor is converted into a first bird's-eye view image through inverse mapping or inverse perspective transformation.

[0046] Each bird's-eye view matching point pair in multiple bird's-eye view matching point pairs refers to a pair of pixels that represent the same physical location or feature, determined within the overlapping area of ​​each first bird's-eye view image and its adjacent first bird's-eye view image in multiple first bird's-eye view images.

[0047] Optionally, feature point extraction is performed on the overlapping area of ​​each first bird's-eye view image and its adjacent first bird's-eye view image in the multiple first bird's-eye view images, and feature points of each pair of adjacent bird's-eye view images in the multiple first bird's-eye view images are matched to obtain multiple pairs of bird's-eye view matching points corresponding to each pair of visual sensors. This can be achieved by: acquiring feature points of each first bird's-eye view image within the overlapping area and generating corresponding feature descriptors; calculating the similarity between the feature descriptors of each first bird's-eye view image and the feature descriptors of adjacent first bird's-eye view images; filtering feature points with matching degrees higher than a preset threshold through feature matching; obtaining the correspondence between feature points of each bird's-eye view image and its adjacent first bird's-eye view images; verifying the geometric consistency of the feature point pairs that establish the correspondence; obtaining verified matching point pairs; and using the pixel coordinates of the matching point pairs in the bird's-eye view coordinate system as bird's-eye view matching points to obtain multiple pairs of bird's-eye view matching points corresponding to each pair of first bird's-eye view images.

[0048] Each pair of visual matching points in the multiple pairs of visual matching points refers to the pixel pair obtained by back-projecting (re-projecting) each pair of bird's-eye view matching points back into the first visual image of each pair of visual sensors. Optionally, each pair of visual matching points in the multiple pairs of visual matching points is located in the coordinate space of the first visual image.

[0049] Optionally, each pair of bird's-eye view matching points corresponding to each pair of visual sensors can be geometrically back-projected back into the first visual image of each pair of visual sensors based on the initial calibration parameters, to obtain multiple visual matching point pairs corresponding to each pair of visual sensors.

[0050] Optionally, mapping each pair of bird's-eye view matching points corresponding to each pair of vision sensors back to the first visual image of each pair of vision sensors to obtain multiple pairs of visual matching points corresponding to each pair of vision sensors may include the following methods: For each pair of bird's-eye view matching points corresponding to each pair of vision sensors, obtain the two-dimensional coordinates of each pair of bird's-eye view matching points in the bird's-eye view coordinate system; based on the initial calibration parameters of each vision sensor, establish a mapping relationship between the bird's-eye view coordinate system and the corresponding first visual image pixel coordinate system; using the mapping relationship, project the two-dimensional coordinates of the bird's-eye view matching points back onto the corresponding first visual image to determine the pixel coordinates of the bird's-eye view matching points in the corresponding first visual image; use the determined pixel coordinates as the visual matching points in the corresponding first visual image to obtain multiple pairs of visual matching points corresponding to each pair of vision sensors.

[0051] The target calibration parameters for each vision sensor refer to the final calibration parameters obtained by correcting the initial calibration parameters of each pair of vision sensors based on multiple pairs of visual matching points corresponding to each pair of vision sensors.

[0052] It should be noted that the target calibration parameters can more accurately reflect the current physical state of the vehicle's vision sensors. The target calibration parameters can correct the initial parameter deviations caused by vehicle bumps, vibrations, or environmental changes, and are used to generate more accurate bird's-eye view images.

[0053] Optionally, a reprojection error objective function is constructed, which includes multiple pairs of visual matching points corresponding to each pair of visual sensors. The reprojection error objective function is used to measure the geometric deviation between the visual matching points back-projected to the bird's-eye view coordinate system based on the initial calibration parameters and the original bird's-eye view matching points. With the goal of minimizing the reprojection error objective function, an iterative optimization algorithm is used to adjust the initial calibration parameters of each pair of visual sensors. When the reprojection error objective function meets the preset convergence condition, the iteration is stopped, and the calibration parameters after the iteration are determined as the target calibration parameters for each visual sensor.

[0054] The second visual image of each visual sensor refers to the original image acquired by each visual sensor after obtaining the target calibration parameters.

[0055] It should be noted that the second visual image can be the first visual image, or it can be a visual image that is re-acquired and is identical to the first visual image.

[0056] The second bird's-eye view image refers to the bird's-eye view image obtained by converting the second visual image using the target calibration parameters of each visual sensor.

[0057] It should be noted that while the first and second bird's-eye view images contain the same content, the first image is generated based on initial calibration parameters. If the visual sensor parameters drift, the generated first bird's-eye view image may exhibit geometric distortion, and the first bird's-eye view images generated by adjacent sensors may show misalignment, ghosting, or discontinuity in overlapping areas. The second bird's-eye view image, on the other hand, is generated based on corrected target calibration parameters, resulting in higher geometric accuracy. The second bird's-eye view images generated by adjacent sensors can achieve better alignment in overlapping areas, reducing ghosting and misalignment. Furthermore, the first bird's-eye view image can be used for feature point extraction and matching within the overlapping area to obtain matching points for the bird's-eye view, supporting the calibration parameter correction process. The second bird's-eye view image is the final output data, used to directly stitch together the final target bird's-eye view image for user viewing or system use.

[0058] The target bird's-eye view image refers to the final complete bird's-eye view panoramic image generated by merging and stitching together multiple second bird's-eye view images according to the spatial layout of the vehicle.

[0059] Optionally, after obtaining the target calibration parameters, a second visual image is acquired by each of the multiple visual sensors. Using the obtained target calibration parameters, the pixels in the second visual image are mapped to the bird's-eye view coordinate system to generate a calibrated bird's-eye view image (second bird's-eye view image). Finally, these calibrated second bird's-eye view images are fused and stitched together according to their spatial positions to generate a seamless, ghost-free, complete bird's-eye view of the vehicle's surrounding environment, i.e., the target bird's-eye view image.

[0060] The embodiments provided in this application acquire a first visual image from each of the multiple visual sensors on a vehicle. Based on the initial calibration parameters of each visual sensor, the first visual image of each visual sensor is converted into a bird's-eye view image, resulting in multiple first bird's-eye view images. This causes adjacent visual sensors to form overlapping regions under the bird's-eye view perspective. Feature points are then extracted and matched from the overlapping regions of adjacent first bird's-eye view images to obtain multiple corresponding bird's-eye view matching point pairs. Each bird's-eye view matching point pair is then back-mapped back to the corresponding first visual image, resulting in multiple visual matching point pairs. Finally, the initial calibration parameters of each visual sensor are corrected based on these multiple visual matching point pairs to obtain target calibration parameters. By utilizing the consistency constraints of multiple pairs of visual matching points in the overlapping area under multiple perspectives, geometric relationships can be established in the original image space through an inverse mapping mechanism. This directly optimizes the calibration parameters to eliminate reprojection errors, thereby correcting the deviations caused by the initial calibration. Finally, based on the corrected target calibration parameters, the second visual image of the visual sensor is mapped to a second bird's-eye view image and stitched together to form the target bird's-eye view image. This allows the generated bird's-eye view to accurately reflect the real geometric relationships of the vehicle's surrounding environment, improving the accuracy of the bird's-eye view. This can solve the technical problem in related technologies where the use of fixed calibration parameters during image generation leads to significant errors between the generated bird's-eye view and the actual scene of the vehicle, resulting in inaccurate bird's-eye views.

[0061] In one exemplary embodiment, based on the initial calibration parameters of each visual sensor, the first visual image of each visual sensor is converted into a bird's-eye view image to obtain multiple first bird's-eye view images. This includes: performing the following conversion operation on each visual sensor as the current visual sensor, wherein the initial calibration parameters of the current visual sensor are the current initial calibration parameters, and the first visual image of the current visual sensor is the current first visual image; based on a first mapping relationship, mapping the bird's-eye view pixel coordinates of each bird's-eye view pixel in the canvas of the current bird's-eye view image to world coordinates in the world coordinate system, to obtain the world coordinates corresponding to each bird's-eye view pixel. The process involves defining a bird's-eye view image canvas, which is a digital image created for the current first visual image. The first mapping relationship is the mapping between the pixel coordinates of the bird's-eye view pixels in the current bird's-eye view image canvas and the world coordinates in the world coordinate system. Based on the current initial calibration parameters, the world coordinates corresponding to each bird's-eye view pixel are converted to pixel coordinates in the pixel coordinate system of the current first visual image, resulting in the original image pixel coordinates corresponding to each bird's-eye view pixel. The pixel value of the original image pixel coordinates corresponding to each bird's-eye view pixel is then assigned to each bird's-eye view pixel in the current bird's-eye view image canvas, resulting in the bird's-eye view image corresponding to the current first visual image.

[0062] In this embodiment, the current visual sensor refers to the visual sensor currently undergoing the conversion operation. The current initial calibration parameters refer to the initial calibration parameters corresponding to the current visual sensor. The current first visual image refers to the original visual image acquired by the current visual sensor.

[0063] The first mapping relationship refers to the mapping relationship between the pixel coordinates of the bird's-eye view pixels in the current bird's-eye view image canvas and the world coordinate system. Optionally, the first mapping relationship can be determined by the physical size and resolution of the current bird's-eye view image canvas.

[0064] Each bird's-eye view pixel refers to the position of each pixel in the current bird's-eye view image canvas, arranged in a pixel matrix.

[0065] The current bird's-eye view image canvas refers to the digital image created for the current first-view image. Optionally, the current bird's-eye view image canvas can be a two-dimensional array area used to store the bird's-eye view data to be generated. The image resolution and physical extent of the current bird's-eye view image canvas are predefined (e.g., 256x256 pixels, corresponding to a 30m x 30m area in the real world).

[0066] The world coordinate system is a coordinate system with the vehicle as its origin. Optionally, the origin of the world coordinate system can be any point on the vehicle (such as the center of the rear axle or the center of the front axle). The X-axis of the world coordinate system points forward of the vehicle, the Y-axis points to the left of the vehicle's direction of travel, and the Z-axis points to the sky.

[0067] The world coordinates corresponding to each pixel in the bird's-eye view refer to the three-dimensional spatial coordinates corresponding to each pixel after transforming its pixel coordinates on the current bird's-eye view image canvas to the world coordinate system of the vehicle, based on the first mapping relationship. Optionally, the world coordinates corresponding to each pixel in the bird's-eye view can be represented as (Xw, Yw, 0).

[0068] Optionally, an empty digital image matrix (i.e., the current bird's-eye view image canvas) is created, which has a specific resolution (e.g., 256x256 pixels) and physical size (e.g., covering an area 15 meters in front of, behind, to the left and right of the vehicle). For each pixel on the current bird's-eye view image canvas (u bev v bev Using a pre-defined first mapping relationship, the three-dimensional physical coordinates (X, X, Z) of each pixel in the world coordinate system (usually the vehicle coordinate system, Z=0 plane) of the vehicle are calculated. w Y w ,0). Here, because the ground in the world coordinate system is assumed to be flat (Z=0), the two-dimensional image pixels are directly associated with the ground position in three-dimensional space.

[0069] The pixel coordinates in the current first visual image's pixel coordinate system refer to the two-dimensional coordinates of the target point obtained in the current first visual image's pixel coordinate system. Optionally, the pixel coordinates in the current first visual image's pixel coordinate system can be represented as (u fisheye v fisheye The original image pixel coordinates corresponding to each bird's-eye view pixel refer to the pixel coordinates of each bird's-eye view pixel in the pixel coordinate system of the current first visual image. That is, the original image pixel coordinates corresponding to each bird's-eye view pixel are also represented as (u fisheye v fisheye ).

[0070] The pixel value corresponding to the original image pixel coordinates of each pixel in the bird's-eye view refers to the color or grayscale information obtained in the current first visual image.

[0071] Optionally, the pixel coordinates in the pixel coordinate system of the current first visual image are usually floating-point numbers and may fall between the pixel grid points of the current first visual image. An interpolation algorithm (such as bilinear interpolation) can be used to calculate the color value at the floating-point coordinates based on the color values ​​of the surrounding pixels in the current first visual image. If the calculated coordinates exceed the boundary of the original image, or the depth Zc of the corresponding sensor coordinate system is ≤0 (indicating that the point is behind the camera and is not visible), the BEV pixel is marked as invalid (e.g., filled with black or transparent). Finally, the calculated color value is filled into each bird's-eye view pixel corresponding to the current bird's-eye view image canvas to obtain the bird's-eye view image corresponding to the current first visual image.

[0072] Optionally, using the current initial calibration parameters (such as the rotation matrix R and translation vector t), point P in the world coordinate system is... w =[X w Y w [0, 1] T Transform to the current vision sensor's coordinate system to obtain point P in the sensor coordinate system. c =[X c Y c Z c ] T Using intrinsic parameters (such as focal length fx, fy, principal point cx, cy) and fisheye distortion models (such as the Mei model or Brown-Conrady model) from the initial calibration parameters, the point Pc in the sensor coordinate system is projected onto the two-dimensional image plane. First, the point is normalized to a unit sphere or normalized plane. Then, radial and tangential distortion coefficients (k1, k2, ... and p1, p2) are applied for distortion correction / simulation. Finally, the theoretical pixel coordinates (u) of this point in the original fisheye image (i.e., the first visual image) are calculated. fisheye v fisheye ).

[0073] In some embodiments, for each pixel on the BEV canvas (i.e., the current bird's-eye view image canvas, a two-dimensional digital image grid (or matrix) used to generate the bird's-eye view), the corresponding pixel on the BEV canvas in the original fisheye image (the image captured by the fisheye lens (i.e., the visual sensor), also known as the first visual image) is found, and then the color information is copied over. This process is called inverse mapping or inverse perspective transformation. Inverse mapping is used instead of first projecting the original image onto the BEV to avoid creating holes and jagged edges in the BEV image, thus ensuring image quality. It is assumed that the ground is flat and that all relevant objects (such as vehicles, pedestrians, etc.) have a height of 0, meaning they are all close to the ground. This assumption simplifies the mapping relationship from the 3D world to the 2D image, resulting in the pixel (u) on the BEV canvas. bev v bevThe coordinates (u) on the original fisheye image corresponding to ) fisheye ,v fisheye ), because (u fisheye v fisheye The value is usually a floating-point number. An interpolation algorithm (such as bilinear interpolation) is needed to calculate the pixel value at that location from the original fisheye image. Then, this calculated pixel value (color) is assigned to the (u) value on the BEV canvas. bev v bev ) position. If the calculated (u fisheye vf isheye If a pixel exceeds the range of the original fisheye image (i.e., the first visual image), it is marked as invalid (e.g., filled with black). If Z_c <= 0, it means the pixel is behind the camera and not visible, and is also marked as invalid. This completes the BEV image. Figure 3 This is a bird's-eye view image obtained from two adjacent visual sensors according to an embodiment of this application, such as... Figure 3 The image shown is an example of a BEV image from the front camera (i.e., the vision sensor at the front of the vehicle) and the left camera (i.e., the vision sensor on the left side of the vehicle's direction of travel).

[0074] In this embodiment, a linear mapping between the pixel coordinates of the bird's-eye view and the physical location in the world coordinate system is established. By using the initial calibration parameters for back projection and pixel value filling, the transformation from the first visual image to the first bird's-eye view image is realized without eliminating the stretching and deformation of the image edges to generate a visually continuous bird's-eye view.

[0075] In an exemplary embodiment, the world coordinate system is the vehicle coordinate system of the vehicle, and the origin of the world coordinate system is a designated point on the vehicle; before mapping the bird's-eye view pixel coordinates of each bird's-eye view pixel in the current bird's-eye view image canvas to world coordinates in the world coordinate system based on the first mapping relationship, the above method further includes: obtaining a preset bird's-eye view area and a preset image resolution, wherein the preset bird's-eye view area is a region within the bird's-eye view plane of the world coordinate system set for the bird's-eye view image to be converted from the current first visual image, and the preset image resolution is an image resolution set for the bird's-eye view image to be converted from the current first visual image; and determining the first mapping relationship based on the region size of the preset bird's-eye view area and the preset image resolution.

[0076] In this embodiment, the world coordinate system is the vehicle coordinate system of the vehicle. The origin of the world coordinate system is a designated point on the vehicle. Optionally, the designated point on the vehicle can be the rear axle center, the front axle center, etc., without specific limitations.

[0077] The preset bird's-eye view area refers to the area within the bird's-eye view plane of the world coordinate system that is set for the bird's-eye view image to be converted from the current first visual image. Optionally, the preset bird's-eye view area can be a predefined two-dimensional rectangular physical area on the ground plane (Z=0) of the vehicle coordinate system for generating the bird's-eye view. The preset bird's-eye view area can determine how much range around the vehicle can be seen through the bird's-eye view.

[0078] For example, the preset bird's-eye view area can be 15 meters in front of the vehicle, 15 meters behind the vehicle, 15 meters to the left of the vehicle, and 15 meters to the right of the vehicle, or it can be 10 meters in front of the vehicle, 10 meters behind the vehicle, 15 meters to the left of the vehicle, and 15 meters to the right of the vehicle.

[0079] The preset image resolution refers to the image resolution set for the bird's-eye view image to be converted from the current first-view image. Optionally, the preset image resolution may include pixel width and pixel height. For example, the preset image resolution may be 256×256 pixels, or 720×720 pixels, etc., which can be set according to the actual situation, and no specific restrictions are imposed here.

[0080] For example, define a preset bird's-eye view area in the world coordinate system (usually also the vehicle coordinate system, with the origin at the center of the vehicle's rear axle). The X-axis of the world coordinate system points forward of the vehicle; the Y-axis points to the left of the vehicle; and the Z-axis points to the sky. Define a rectangular area on this XY plane as the BEV image range, where the X range is from -15 meters to 15 meters (15 meters behind the vehicle to 15 meters in front); and the Y range is from -15 meters to 15 meters (15 meters to the right and left of the vehicle). Then, create an empty digital image (BEV canvas). The pixels of this canvas need to be correlated with the world coordinates. Assume the BEV image resolution is W. bev ×H bev Pixel, each pixel (u bev , v bev ) corresponds to a world coordinate (X) w Y w The mapping relationship (0), is determined by the physical size of the BEV image (i.e., the preset bird's-eye view area) and the preset image resolution. For example, for the X direction: X w =Xmin + (u bev + 0.5) (Xmax - Xmin) / W bev Where Xmax is the maximum physical coordinate value of the preset bird's-eye view area in the X-axis direction, and Xmin is the minimum physical coordinate value of the preset bird's-eye view area in the X-axis direction. For the Y direction: Y w = Ymin+ (v bev + 0.5) (Ymax - Ymin) / H bev Where Ymax is the maximum physical coordinate value of the preset bird's-eye view area in the Y-axis direction, and Ymin is the minimum physical coordinate value of the preset bird's-eye view area in the Y-axis direction, (u bev +0.5) is to get the center point of the pixel, not the corner, which is more accurate. (Xmax - Xmin) / W bev It is the physical size (meters / pixel) that each pixel of the BEV image represents in the X direction.

[0081] Optionally, determining the first mapping relationship based on the area size of the preset bird's-eye view region and the preset image resolution may include the following methods: Based on the physical size of the preset bird's-eye view region along a preset coordinate axis and the number of pixels in the corresponding pixel axis direction of the preset image resolution, obtain the physical distance represented by each pixel unit as a scaling factor; based on the width and height of the preset image resolution, determine the center pixel coordinates of the center point of the preset bird's-eye view region in the pixel coordinate system, obtain the theoretical world coordinates corresponding to the center pixel coordinates, acquire the actual set coordinates of the center point of the preset bird's-eye view region in the world coordinate system, and determine the difference between the theoretical world coordinates and the actual set coordinates corresponding to the center pixel coordinates as the offset; based on the scaling factor and the offset, construct a linear mapping equation between the bird's-eye view pixel coordinates and the world coordinates to obtain the first mapping relationship.

[0082] In this embodiment, the world coordinate system is defined as the vehicle coordinate system with a designated point of the vehicle as the origin. By combining the preset physical view area and image resolution to determine the mapping relationship, each pixel in the bird's-eye view strictly corresponds to the specific physical size in the real world. This ensures the geometrical authenticity of the bird's-eye view and gives physical meaning to distance measurements in the image. By using preset areas and resolutions, the field of view around the vehicle and the image clarity covered by the bird's-eye view can be flexibly controlled to adapt to the needs of different scenarios for field of view size and resolution, ensuring the consistency and standardization of the generated images.

[0083] In an exemplary embodiment, the current initial calibration parameters include current initial calibration extrinsic parameters; based on the current initial calibration parameters, converting the world coordinates corresponding to each bird's-eye view pixel into pixel coordinates in the pixel coordinate system of the current first visual image to obtain the original image pixel coordinates corresponding to each bird's-eye view pixel includes: based on the current initial calibration extrinsic parameters, converting the world coordinates corresponding to each bird's-eye view pixel into local coordinates in the sensor coordinate system of the current visual sensor to obtain the sensor local coordinates corresponding to each bird's-eye view pixel; using the calibration intrinsic parameters of the current visual sensor and the sensor distortion model of the current visual sensor, converting the sensor local coordinates corresponding to each bird's-eye view pixel into pixel coordinates in the pixel coordinate system of the current first visual image to obtain the original image pixel coordinates corresponding to each bird's-eye view pixel.

[0084] In this embodiment, the current initial calibration extrinsic parameters refer to the parameters of the current position and orientation of the vision sensor in the world coordinate system. Optionally, the current initial calibration extrinsic parameters may include three translation parameters (displacement in the X, Y, and Z directions) and three rotation parameters (Roll, Pitch, and Yaw angles).

[0085] The current sensor coordinate system refers to a local three-dimensional coordinate system with the optical axis of the vision sensor as the origin, the Z-axis pointing in the direction of the optical axis of the vision sensor (usually forward), and the X-axis and Y-axis parallel to the horizontal and vertical directions of the image plane, respectively. Optionally, in the sensor coordinate system, the vision sensor is stationary, and objects in the world move relative to the vision sensor.

[0086] The local coordinates in the sensor coordinate system of the current vision sensor refer to the coordinates of a pixel in the sensor coordinate system. Optionally, the local coordinates in the sensor coordinate system of the current vision sensor can be represented as Pc = [Xc, Yc, Zc]. T express.

[0087] The calibration intrinsic parameters of the current vision sensor refer to the parameters of the optical characteristics of the current vision sensor. Optionally, the calibration intrinsic parameters of the current vision sensor may include the focal length and principal point (the coordinates of the intersection of the optical axis and the image plane in the pixel coordinate system).

[0088] The sensor distortion model of current vision sensors refers to the mathematical model used to describe the optical distortion of current vision sensors (such as fisheye lenses). Optionally, sensor distortion models include the Mei model (based on unit sphere projection), the Kannala-Brandt model, etc.

[0089] Optionally, taking a fisheye lens as an example of a current vision sensor, due to the large field of view of a fisheye lens, light will bend non-linearly when passing through the lens, causing straight lines to become curved. The sensor distortion model corrects this non-linear deformation through polynomial coefficients (such as radial distortion coefficient and tangential distortion coefficient).

[0090] In one embodiment, a point P in the BEV world coordinate system is provided. w = [Xw, Yw, 0, 1] T It is necessary to transform P_w to the camera coordinate system using the initial calibration parameters (such as extrinsic parameters) of the current vision sensor. The extrinsic parameters describe the relative relationship between the world coordinate system and the camera coordinate system, including the rotation matrix R (a 3x3 matrix) and the translation matrix t (a 3x1 vector). The transformation formula (1) can be:

[0091]

[0092] Specifically, it can be written in the form of the following formula (2):

[0093]

[0094] Because of P w Since the Z component is 0, the above equation (2) can be simplified to the following formula (3):

[0095]

[0096] Where P c = [X c Y c Z c ] T It is point P w The coordinates Z in the sensor coordinate system c This represents the depth from the point to the optical center of the vision sensor and is a key denominator in subsequent projection calculations. Then, projecting from the sensor coordinate system to the fisheye image pixel coordinate system (i.e., the pixel coordinate system of the first visual image) requires using the calibration intrinsic parameters of the vision sensor (such as a camera) and a fisheye distortion model (such as the Mei fisheye distortion model). First, point P in the sensor coordinate system... c Normalization yields the coordinates of the point on the unit sphere, which can be obtained using the following formula (4):

[0097]

[0098] Where r = sqrt(X_c^2 + Y_c^2 + Z_c^2) is the value of point P. c The distance to the optical center of the vision sensor.

[0099] Projecting the above equation (4) onto the virtual normalized plane, we can obtain it using the following formula (5):

[0100]

[0101] The parameter ξ(xi) controls the characteristics of the projection. When ξ = 0, the model degenerates into a standard perspective projection. When ξ > 0, the model can describe the wide-angle characteristics of a fisheye camera.

[0102] Based on the above virtual normalized plane coordinates, the following formulas (6) and (7) can be obtained:

[0103]

[0104]

[0105] Where k1, k2, k3, and k4 are all preset values. Based on the above formulas (5) and (7), the radial distortions of the following formulas (8) and (9) can be obtained:

[0106]

[0107]

[0108] Based on the above formulas (5), (8) and (9), the tangential distortions of the following formulas (10) and (11) can be obtained:

[0109]

[0110]

[0111] Based on formulas (10) and (11), the final pixel coordinates of formulas (12) and (13) can be obtained:

[0112]

[0113]

[0114] This yields the original image pixel coordinates corresponding to each pixel in the bird's-eye view. , ).

[0115] In this embodiment, based on the current initial calibration extrinsic parameter transformation and intrinsic parameter distortion projection, a mathematical mapping is established from the physical space of the bird's-eye view to the pixel space of the first visual image. This enables the accurate backtracking of any point on the BEV plane to the corresponding position in the first visual image. By introducing a sensor distortion model, the radial and tangential distortions unique to the visual sensor are compensated during the projection process, which can significantly improve the accuracy of the reverse mapping. This ensures that the corresponding pixel points can be found more accurately in the edge region of the first visual image, thereby guaranteeing the geometric consistency of the bird's-eye view stitching edges.

[0116] In an exemplary embodiment, a feature point of each first bird's-eye view image is represented by a feature point position and a descriptor vector. Matching feature points of each pair of adjacent bird's-eye view images in a plurality of first bird's-eye view images to obtain multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors includes: performing the following feature point matching operation on each pair of bird's-eye view images as the current pair of bird's-eye view images to obtain multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors, wherein the current pair of bird's-eye view images includes a first current bird's-eye view image and a second current bird's-eye view image: embedding and encoding the descriptor vector of each feature point in the first current bird's-eye view image, and positionally encoding the feature point position of each feature point in the first current bird's-eye view image to obtain a first current feature sequence corresponding to the first current bird's-eye view image, and the pair of visual sensors corresponding to the current pair of bird's-eye view images is the current pair of visual sensors; for the second current... The descriptor vector of each feature point in the bird's-eye view image is embedded and encoded, and the feature point position of each feature point in the second current bird's-eye view image is positionally encoded to obtain the second current feature sequence corresponding to the second current bird's-eye view image. The first current feature sequence and the second current feature sequence are input into an M-layer decoder, and the set of multiple feature point pairs indicated by the matching results output by each layer decoder is determined as a candidate matching point pair set. Each layer decoder is used to perform cross-attention matching on the two input feature sequences and outputs indication information of feature point pairs with a matching confidence greater than or equal to a preset confidence threshold. M is a positive integer greater than or equal to 1. Multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors are selected from the candidate matching point pair set, wherein the number of bird's-eye view matching point pairs in the multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors is less than or equal to a preset number.

[0117] In this embodiment, a feature point of each first bird's-eye view image is a key pixel or point of interest that has significant visual information and is located stably, determined from the overlapping area of ​​two adjacent first bird's-eye view images.

[0118] It should be noted that not every pixel in every first bird's-eye view image is a feature point. Only pixels with unique textures, edges, corners, or high-contrast areas in each first bird's-eye view image can be identified as feature points.

[0119] The feature point location is the two-dimensional pixel coordinates of the feature point in the first bird's-eye view image. Optionally, the feature point location can provide spatial geometric information of the feature point.

[0120] A descriptor vector is a multidimensional numerical vector used to uniquely identify the visual features of a local region in an image. Optionally, the descriptor vector can be calculated by a feature extraction network (such as SuperPoint) and can be used to characterize the local appearance features (such as texture, gradient, color, etc.) of feature points.

[0121] The current pair of bird's-eye view images refers to the pair of bird's-eye view images for which feature point matching is currently performed. Optionally, in a surround-view system, bird's-eye view images generated by adjacent visual sensors (such as the left camera and the front camera) may have overlapping areas in space. The bird's-eye view images corresponding to these overlapping areas are extracted to obtain the current pair of bird's-eye view images.

[0122] Here, the current bird's-eye view image includes a first current bird's-eye view image and a second current bird's-eye view image.

[0123] The first current bird's-eye view image refers to the bird's-eye view image identified as the source image in the current pair of bird's-eye view images, and the second current bird's-eye view image refers to the bird's-eye view image identified as the target image in the current pair of bird's-eye view images. For example, when the current pair of bird's-eye view images is a pair of bird's-eye view images obtained by the vehicle's front vision sensor and left vision sensor, the first current bird's-eye view image can be the bird's-eye view image generated by the front vision sensor, and the second current bird's-eye view image can be the bird's-eye view image generated by the left vision sensor; or, the first current bird's-eye view image can be the bird's-eye view image generated by the left vision sensor, and the second current bird's-eye view image can be the bird's-eye view image generated by the front vision sensor.

[0124] Embedding encoding refers to the process of mapping raw data (such as descriptors) to a continuous, high-dimensional vector space through linear or nonlinear transformations. Optionally, embedding encoding can help models better capture the semantic information of features.

[0125] Positional encoding is the process of encoding the positional information of input elements (such as pixel coordinates in an image) into a vector and adding it to the feature vector, enabling the model to utilize spatial positional information.

[0126] The first current feature sequence refers to the linearly ordered vector sequence of all feature points in the first current bird's-eye view image, formed after embedding and positional encoding processing. Optionally, each element in the first current feature sequence represents a feature point, and each element includes the embedded descriptor vector and the feature point position.

[0127] The second current feature sequence refers to the linearly ordered vector sequence of all feature points in the second current bird's-eye view image, formed after embedding and positional encoding processing. Optionally, each element in the second current feature sequence represents a feature point, and each element includes the embedded descriptor vector and the feature point position.

[0128] An M-layer decoder refers to a structure based on a neural network module, obtained by stacking M identical decoder layers, where M is a positive integer greater than or equal to 1. Optionally, each decoder layer in the M-layer decoder includes a self-attention mechanism, a cross-attention mechanism, and a feedforward neural network. In matching tasks, the M-layer decoder is used to aggregate contextual information and calculate the correlation between feature points in different images. For example, the M-layer decoder can be a Transformer decoder, a recurrent neural network decoder, or a contrastive learning decoder.

[0129] The matching result output by each decoder layer refers to the matching confidence or matching indication information output after processing by each decoder layer. Specifically, each decoder layer can perform cross-attention matching on two input feature sequences and output indication information for feature point pairs whose matching confidence is greater than or equal to a preset confidence threshold. Optionally, each decoder layer can perform cross-attention matching on a first current feature sequence and a second feature sequence and output indication information for feature point pairs whose matching confidence is greater than or equal to a preset confidence threshold.

[0130] The preset confidence threshold refers to a pre-defined confidence level. Optionally, the preset confidence threshold can be 0.95 or 0.90. For example, a confidence threshold (i.e., the preset confidence threshold) of 0.95 means that the algorithm has a high probability of finding the correct set of interior points, thus balancing computational efficiency and estimation accuracy.

[0131] Cross-attention is an attention mechanism that allows one sequence (such as features of image A) to focus on another sequence (such as features of image B). Optionally, in matching, cross-attention can calculate the similarity weight between a point in image A and all points in image B, thereby achieving point-to-point association.

[0132] The candidate matching point pair set refers to the set of multiple feature point pairs indicated by the matching results output by the layer decoder. Optionally, the candidate matching point pair set includes all feature point pairs in layers 1 to M that satisfy a matching confidence level greater than or equal to a preset confidence threshold.

[0133] Optionally, the method for selecting multiple bird's-eye view matching point pairs corresponding to the current visual sensor from the candidate matching point pair set may include: sorting the bird's-eye view matching point pairs in the candidate matching point pair set from high to low according to their matching confidence, and selecting bird's-eye view matching point pairs in descending order of matching confidence until the number of selected bird's-eye view matching point pairs reaches a preset number, thus obtaining multiple bird's-eye view matching point pairs; or sorting the candidate matching point pair set from high to low according to their matching confidence; initializing an empty target matching point pair set, and marking all pixel regions as unoccupied. The process involves iterating through the sorted candidate matching point pairs. For the current bird's-eye view matching point pair to be selected, it is determined whether the position of the feature point corresponding to the current bird's-eye view matching point pair in the bird's-eye view image is located in an occupied area. If the position is located in an unoccupied area, the bird's-eye view matching point pair is added to the target matching point pair set, and the position of the bird's-eye view matching point pair is marked as occupied. When the number of matching point pairs in the target matching point pair set reaches a preset number, or when the traversal ends, the matching point pairs in the target matching point pair set are determined as the current multiple bird's-eye view matching point pairs corresponding to the visual sensor.

[0134] In some embodiments, BEV images from multiple visual sensors (such as surround-view cameras) naturally overlap in the area surrounding the vehicle, for example, as Figure 3As shown, BEV matching points are obtained by matching overlapping region images, which can be performed using Lightglue. First, the images are preprocessed by converting the input BEV image pairs into grayscale images and normalizing them to the range [0,1]. Then, the SuperPoint feature detector is used. Assuming there are two images I^A and I^B to be matched, SuperPoint (a lightweight feature and descriptor generation algorithm based on deep learning) generates a set of outputs for each image: keypoint locations (i.e., feature point locations) and descriptor vectors. These raw descriptor vectors, while containing rich visual information, have not yet been optimized for the matching task. The keypoint locations (i.e., feature point locations) and descriptor vectors will serve as the initial input for subsequent steps. Next, feature extraction and encoding are performed. The goal of this stage is to transform the raw, independent descriptor vectors into more powerful feature representations containing contextual information. A multilayer perceptron (MLP) with shared weights is used to perform preliminary embedding on each descriptor vector, mapping it to a hidden space more suitable for Transformer processing. However, relying solely on appearance descriptor vectors is insufficient. To help the model understand the geometric structure of the image, positional encoding is introduced. This is a standard operation in the Transformer architecture, designed to provide the model with positional information of elements in the sequence. Since keypoints are two-dimensional, standard sine and cosine functions are used to generate the positional encoding e_i. At this point, two sets of enhanced feature sequences (i.e., the first current feature sequence and the second current feature sequence) are obtained for the two images respectively. These two current feature sequences are then fed into the core Transformer module. Then, based on Transformer cross-attention matching, through a series of Transformer decoder layers, the features of the two images fully "exchange information," thereby learning whether each keypoint (i.e., feature point) should be matched, and with whom. Each Transformer layer contains two core sub-modules: self-attention and cross-attention. Self-attention allows information exchange between keypoints within an image. For example, a keypoint (i.e., feature point) can learn through the self-attention mechanism what other keypoints are around it, and what the feature point looks like, thus understanding itself within a larger context. Cross attention is key to enabling information exchange between images. Cross attention allows each key point (i.e., feature point) in image A to "query" all key points (i.e., feature points) in image B, and vice versa. Through this process, a key point (i.e., feature point) can collect information about all possible candidate matching points in another image.Finally, predictable matching decisions are set up by attaching a classifier to each Transformer layer, enabling it to "think ahead" and exit early when it is confident. The model will identify those point pairs that have been judged as matches with high confidence, as well as those points that have been judged as unmatched with high confidence, and finally output the matching results (i.e., obtain multiple bird's-eye view matching point pairs corresponding to the current visual sensor). Figure 4 This is a schematic diagram of an optional BEV image matching result according to an embodiment of this application, such as... Figure 4 As shown, this represents the BEV image matching point results for the front and left lenses.

[0135] In this embodiment, the matching result is dynamically determined based on confidence level, thereby reducing unnecessary computational resource consumption and meeting the real-time requirements of the vehicle system. By utilizing the cross-attention mechanism, the model can capture the global contextual dependencies between image point pairs, rather than relying solely on local appearance similarity, thus effectively suppressing noise and mismatches and improving the accuracy of the matching results. At the same time, through multi-level feature interaction and confidence level filtering, the matching quality is gradually optimized. By setting a preset quantity constraint, the number of output matching point pairs is ensured to be within a controllable range. This avoids the problem of excessive subsequent geometric optimization computation burden or getting trapped in local optima due to too many matching points, ensuring the stability and convergence speed of the self-calibration algorithm. By combining the appearance information of the descriptor vector and the geometric information of the position encoding, the matching process considers not only visual similarity but also spatial rationality, thereby obtaining higher quality matching point pairs in complex scenes.

[0136] In one exemplary embodiment, selecting multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors from the candidate matching point pair set includes: if the number of candidate matching point pairs in the candidate matching point pair set is less than or equal to a preset number, selecting all candidate matching point pairs in the candidate matching point pair set as multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors; if the number of candidate matching point pairs in the candidate matching point pair set is greater than the preset number, performing a consistency check on each candidate matching point pair in the candidate matching point pair set; removing candidate matching point pairs that fail the consistency check from the candidate matching point pair set to obtain an updated candidate matching point pair set; and selecting all candidate matching point pairs in the updated candidate matching point pair set as multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors.

[0137] Feature point matching algorithms (such as deep learning-based descriptor matching methods) are commonly used to extract feature points within the overlapping regions of adjacent camera fields of view, forming a set of candidate matching point pairs. However, the candidate matching point selection mechanism in the above methods has the following problems: in complex scenes (such as sparse textures, varying lighting, and interference from dynamic objects), a large number of outliers are easily generated. Related techniques often rely on fixed thresholds or simple geometric constraints, which are difficult to adapt to the changing scene features, resulting in the retained matching point pairs still containing a lot of noise, which in turn affects the convergence and accuracy of subsequent calibration and optimization models. To address the issue that fixed thresholds or simple geometric constraints can lead to significant noise in the remaining matching point pairs, thus affecting the convergence and accuracy of subsequent calibration and optimization models, this embodiment first determines the relationship between the number of candidate matching point pairs and a preset number: when the number of candidate matching point pairs is less than or equal to the preset number, all candidate matching point pairs are directly selected as the final matching point pairs to avoid unnecessary complex calculations when data is sufficient, thereby improving efficiency; when the number of candidate matching point pairs is greater than the preset number, consistency checks are performed on each candidate matching point pair, and outliers that fail the check are removed, retaining high-confidence matching point pairs. This allows for full utilization of limited matching information in feature-sparse scenarios; in feature-rich scenarios, a filtering mechanism ensures data quality, thereby maintaining stable self-calibration performance in various complex environments.

[0138] In this embodiment, if the number of candidate matching point pairs in the candidate matching point pair set is less than or equal to a preset number, all candidate matching point pairs in the candidate matching point pair set are selected as multiple bird's-eye view matching point pairs corresponding to the current visual sensor. For example, if the preset number is 5 and the number of candidate matching point pairs in the candidate matching point pair set is 4, satisfying the condition that the number of candidate matching point pairs in the candidate matching point pair set is less than the preset number, then all candidate matching point pairs in the candidate matching point pair set are determined as multiple bird's-eye view matching point pairs corresponding to the current visual sensor.

[0139] Consistency verification refers to a verification method based on geometric or statistical logic. Optionally, consistency verification can utilize the geometric transformation model (such as homography matrix, affine transformation matrix, or fundamental matrix) determined by the majority of candidate matching point pairs in the current candidate matching point pair set to calculate the geometric error (such as reprojection error or distance error) of each candidate matching point pair. For example, consistency verification can include RANSAC-based consistency verification, homography-based reprojection error consistency verification, or consistency verification based on local geometry (such as DenseCorrespondence).

[0140] Optionally, if the number of candidate matching point pairs in the candidate matching point pair set is greater than a preset number, performing consistency verification on each candidate matching point pair in the candidate matching point pair set may include the following methods: taking the feature points in the first current bird's-eye view image as the center, obtaining local image patches within a first preset radius; in the second current bird's-eye view image, taking the position of the matching point corresponding to the feature point in the first current bird's-eye view image as the center, obtaining local image patches of the same size within a second preset radius; calculating the similarity between the local image patches within the first preset radius and the local image patches within the second preset radius; determining whether the similarity is greater than a preset similarity threshold; if the similarity is greater than or equal to the preset similarity threshold, then determining that the consistency verification of the candidate matching point pair has passed; otherwise, determining that the consistency verification of the candidate matching point pair has failed.

[0141] Optionally, candidate matching pairs that fail the consistency check in the candidate matching pair set can be identified as abnormal matching pairs.

[0142] If there are candidate matching point pairs that fail the consistency check, remove the candidate matching point pairs that fail the consistency check from the candidate matching point pair set to obtain an updated candidate matching point pair set; select all candidate matching point pairs in the updated candidate matching point pair set as multiple bird's-eye view matching point pairs corresponding to the current visual sensor.

[0143] This embodiment determines the number of candidate matching point pairs in the candidate matching point pair set and dynamically selects a processing strategy. When the number of candidate matching points is less than the preset number, they are directly retained to avoid over-screening and loss of effective information. When the number of candidate matching points is greater than the preset number, they are filtered to prevent excessive computation. Furthermore, through consistency verification, outliers with high matching confidence but unreasonable geometric positions can be effectively eliminated. This ensures that the final input matching point pairs have high consistency in geometric transformation, providing a high-quality data foundation for subsequent extrinsic parameter optimization.

[0144] In one exemplary embodiment, consistency verification is performed on each candidate matching point pair in the candidate matching point pair set, including at least one of the following: determining the reprojection error between the two candidate matching points in each candidate matching point pair to obtain the reprojection error corresponding to each candidate matching point pair; performing consistency verification on each candidate matching point pair based on the reprojection error corresponding to each candidate matching point pair; determining the slope direction of the slope of the line connecting the two candidate matching points in each candidate matching point pair to obtain the slope direction corresponding to each candidate matching point pair; performing consistency verification on each candidate matching point pair based on the slope direction corresponding to each candidate matching point pair; determining the Euclidean distance between the two candidate matching points in each candidate matching point pair to obtain the Euclidean distance corresponding to each candidate matching point pair; and performing consistency verification on each candidate matching point pair based on the Euclidean distance corresponding to each candidate matching point pair.

[0145] Due to factors such as changes in ambient lighting, interference from dynamic objects, and sparse or repetitive textures, the initial set of candidate feature point pairs obtained through matching often contains a large number of outliers. To remove these outliers, a single consistency verification mechanism is usually used, such as a method based on statistical models (e.g., RANSAC) or global geometric constraints. However, this method is difficult to effectively capture the subtle local geometric patterns between adjacent matching points. As a result, when features are dense or noise is high, it is difficult to distinguish between true geometric consistency and random noise, limiting the filtering effect. Furthermore, it ignores the scale rationality of the matching points in physical space. To address the challenge of distinguishing true geometric consistency from random noise due to dense features or high noise levels, this embodiment performs consistency verification on candidate matching point pairs based on at least one dimension: reprojection error, slope direction, or Euclidean distance. By introducing multiple verification dimensions such as reprojection error, slope direction, and Euclidean distance, matching points can be comprehensively evaluated from multiple perspectives, including geometric projection consistency, local structural orientation consistency, and spatial distance rationality. This effectively identifies and eliminates different types of mismatched points, significantly improving the confidence and quality of retained matching point pairs. The most suitable verification dimension can be selected based on the feature distribution of the current scene (such as texture richness, road curvature, etc.), and more refined geometric constraints (i.e., slope direction and Euclidean distance) ensure higher geometric consistency of matching point pairs, reducing noise interference during the optimization process, thereby accelerating the optimization convergence process and improving the accuracy of the final calibration parameters, effectively improving the stitching quality of the panoramic image.

[0146] In this embodiment, reprojection error refers to the pixel distance between the projected point and the actual observed point after a 3D spatial point or 2D image point is projected onto another view through a model (such as a geometric transformation model). Optionally, in consistency verification, reprojection error can be used to quantify the degree of consistency of the geometric relationship between two matching points. The smaller the reprojection error, the more reliable the geometric transformation relationship between the two matching points.

[0147] Optionally, the reprojection error between the two candidate matching points in each candidate matching point pair is determined to obtain the reprojection error corresponding to each candidate matching point pair. Based on the reprojection error corresponding to each candidate matching point pair, the consistency verification method for each candidate matching point pair may include: initializing the number of iterations and the maximum inlier count; within a preset number of iterations, randomly selecting the smallest sample set from the candidate matching point pair set to estimate an initial transformation model; using the initial transformation model, calculating the reprojection error of all candidate matching point pairs in the candidate matching point pair set (for example, using the initial transformation model to map from image A to image B (or vice versa), calculating the distance between the mapped point and the actual observation point to obtain the reprojection error of all candidate matching point pairs); counting the number of candidate matching point pairs with reprojection errors less than a preset error threshold; if the number of such candidate matching point pairs is greater than the maximum inlier count, updating the maximum inlier count, and marking all currently counted candidate matching point pairs with reprojection errors less than the preset error threshold as inliers; after the iteration ends, the candidate matching point pairs marked as inliers are determined to have passed the consistency verification, and the candidate matching point pairs not marked as inliers are determined to have failed the consistency verification.

[0148] In some embodiments, the filtering and screening of BEV matching point pairs includes pre-inspecting the number of candidate matching point pairs and performing RANSAC constraint checking. First, the number of candidate matching point pairs is verified, which requires the number of candidate matching point pairs to be greater than or equal to 4 (the minimum requirement for the RANSAC algorithm to estimate the affine transformation matrix). The affine transformation has 6 degrees of freedom (translations tx, ty, rotation θ, scalings sx, sy, shearing sh), and each candidate matching point pair provides two constraint equations. The selection of 4 candidate matching point pairs is for robustness consideration: although theoretically 3 candidate matching point pairs are sufficient for solving, 4 candidate matching point pairs provide redundancy, so that RANSAC can still obtain a stable solution when some candidate matching points are outliers. The ransacReprojThreshold threshold (i.e., the preset error threshold) of RANSAC is set to 1.8, which is approximately equal to a re-projection error of 2 pixels, corresponding to a physical distance error of about 3-5 cm in the BEV image. A strict threshold ensures that only matching points with extremely high spatial consistency are retained. If the re-projection error of a certain matching point is less than the ransacReprojThreshold threshold, the matching point is considered to conform to the currently estimated model and is marked as an inlier. If the re-projection error is greater than the ransacReprojThreshold threshold, the matching point is considered not to conform to the model and is marked as an outlier (mismatch).

[0149] The slope direction of the connecting line between two candidate matching points in each candidate matching point pair refers to the inclination degree and pointing direction of the straight line connecting the two candidate matching points in the two-dimensional image coordinate system.

[0150] Optionally, the manner of determining the slope direction of the connecting line between two candidate matching points in each candidate matching point pair to obtain the slope direction corresponding to each candidate matching point pair may be: acquiring the coordinates of the two candidate matching points in each candidate matching point pair to obtain a first coordinate (x1, y1) and a second coordinate (x2, y2), and calculating the slope direction of the two candidate matching points through the first coordinate and the second coordinate (for example, the slope direction is represented by calculating an angle or judging a positive / negative sign); counting the distribution of the slope directions of all candidate matching point pairs. If the deviation between the slope direction of a certain candidate matching point pair and the slope directions of most candidate matching point pairs exceeds a preset angle threshold, it is determined that the consistency check of this candidate matching point pair fails; if the deviation between the slope direction of a certain candidate matching point pair and the slope directions of most candidate matching point pairs is less than or equal to the preset angle threshold, it is determined that the consistency check of this candidate matching point pair passes.

[0151] The Euclidean distance between two candidate matching points in each candidate matching point pair refers to the straight-line distance between the two candidate matching points in each candidate matching point pair in Euclidean space.

[0152] Optionally, the coordinates of the two candidate matching points are (x1, y1) and (x2, y2), which can be determined by... Obtain the Euclidean distance for each candidate matching point pair. A preset distance interval is defined. If the calculated Euclidean distance falls within this interval, the consistency check for the candidate matching point pair is considered successful; otherwise, the consistency check is considered unsuccessful.

[0153] In some embodiments, fine-grained filtering based on the motion model can be implemented, considering the characteristics of the matching line, namely slope consistency and distance constraints. For slope consistency, from a BEV perspective, vehicle motion between adjacent frames is primarily horizontal displacement. Correct matching points should exhibit a consistent slope pattern. The number of positive and negative slopes of the matching line (i.e., each candidate matching point pair) is counted, retaining the majority. In cases of a tie, all matches are conservatively discarded, thus obtaining the result after slope consistency filtering. For distance constraints, given a BEV image size of 256×256, the length constraint of the matching line (i.e., each candidate matching point pair) is between 150 and 350 Euclidean distances, avoiding some obvious matching errors. Figure 5 This is an image obtained after an optional consistency check according to an embodiment of this application, such as... Figure 5 As shown, this is an example after filtering and selecting matching points for the BEV images of the front and left lenses.

[0154] This embodiment provides three different verification methods based on reprojection error, slope direction, and Euclidean distance. These methods can filter matching points from multiple perspectives, including geometric transformation consistency, statistical distribution consistency, and physical distance rationality. This allows the system to adapt to different scene characteristics, improving the accuracy and robustness of matching point selection. Verification based on reprojection error and slope direction can directly eliminate mismatched points that, although visually similar, do not conform to the overall transformation or motion patterns, significantly reducing the outlier rate. Thus, the matching point pairs selected through consistency verification have higher consistency, providing higher-quality data input for subsequent extrinsic parameter optimization steps. This reduces the risk of divergence during optimization, improving the convergence speed and final accuracy of the calibration results.

[0155] In one exemplary embodiment, based on multiple visual matching point pairs corresponding to each pair of visual sensors, the initial calibration parameters of each visual sensor are corrected to obtain the target calibration parameters of each visual sensor. This includes: performing multiple rounds of correction operations on the calibration parameters of each visual sensor to obtain the target calibration parameters of each visual sensor. During the current round of correction operations, the initial calibration parameters of each visual sensor before correction are the current calibration parameters of each visual sensor. The set of multiple visual matching point pairs corresponding to each pair of visual sensors is called the visual matching point pair set. Based on the current calibration parameters of each visual sensor, the Euclidean distance of each visual matching point pair in the visual matching point pair set on the bird's-eye view plane is determined to obtain the current residual corresponding to each visual matching point pair. Based on the current residual of each visual matching point pair... Determine the current value of the target loss function; based on the current value of the target loss function, correct the current calibration parameters of each vision sensor to reduce the value of the target loss function; wherein, the value of the target loss function is the sum of the robust loss values ​​corresponding to each visual matching point pair; for the k-th visual matching point pair in the set of visual matching point pairs, when the absolute value of the residual corresponding to the k-th visual matching point pair is less than or equal to a preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the square of the residual corresponding to the k-th visual matching point pair; when the absolute value of the residual corresponding to the k-th visual matching point pair is greater than the preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the product of α, the absolute value of the residual corresponding to the k-th visual matching point pair and the preset threshold, minus the product of β and the preset threshold, where α≥1, β>0.

[0156] Typically, calibration parameters can be corrected using a preset threshold or a fixed weighting strategy to obtain the target calibration parameters. However, these methods often result in significant positional deviations in the matching points, severely distorting the direction of parameter correction and preventing convergence to the true calibration parameters of the visual sensor. This leads to ghosting, misalignment, or geometric distortion in the image. To address this issue, in this embodiment, in each iteration, based on the comparison between the absolute value of the residual of the visual matching point pair and a preset threshold, point pairs with smaller residuals are considered high-quality matches and given higher optimization weights to pursue high-precision convergence. Point pairs with larger residuals are considered potential outliers, and a linearized contribution strategy is used to reduce their interference with the overall optimization direction, thereby suppressing their negative impact on calibration parameter correction. Even in scenarios with significant noise or mismatches, accurate calibration results can be obtained, reducing the occurrence of ghosting, misalignment, or geometric distortion in the image.

[0157] In this embodiment, the calibration parameters of each vision sensor before correction are the current calibration parameters of each vision sensor. Optionally, the current calibration parameters of each vision sensor can be the initial calibration parameters or the calibration parameters obtained after the previous round of correction operations.

[0158] Optionally, an upper limit on the number of iterations or a convergence condition can be set to initiate a multi-round correction loop for the calibration parameters of each vision sensor.

[0159] The current residual for each visual matching point pair is the distance between the corresponding pixel positions of the same visual matching point in bird's-eye views from two different cameras, under the current calibration parameters. Optionally, the current residual for each visual matching point pair measures the geometric consistency error of the current calibration result.

[0160] Optionally, using the current calibration parameters, the two visually matched points in the visually matched point pair are back-projected from the pixel coordinate system of their respective first visual images to an assumed ground plane (world coordinate system Z=0), or one of the visually matched points is transformed to the sensor coordinate system of another visual sensor according to the current calibration parameters, and then projected onto the bird's-eye view plane. The Euclidean distance between the position coordinates of the two visually matched points on the bird's-eye view plane is calculated; this Euclidean distance is defined as the current residual of the visually matched point pair in the current iteration. Ideally, if the current calibration parameters are perfect, the BEV positions corresponding to the same physical point in the two images should coincide, and the residual should be 0.

[0161] For example, the current residual corresponding to each visual matching point pair can be represented as r. K (x) = || P 1k {xy} - P 2k {xy} ||, where P 1k {xy} represents the coordinates of a visual matching point in each pair of visual matching points, obtained by backprojecting the corresponding pixel coordinates of the first visual image onto the assumed ground plane, P. 2k {xy} The coordinates of the other visual matching point in each visual matching point pair are obtained by backprojecting the pixel coordinates of the corresponding first visual image onto the assumed ground plane.

[0162] The target loss function is the sum of the robust loss values ​​for each visual matching point pair. The current value of the target loss function refers to the sum of the robust loss values ​​for all visual matching point pairs determined based on the current calibration parameters in the current iteration. Optionally, the current value of the target loss function can be used as a numerical indicator to measure the accuracy of the current calibration parameters. The smaller the current value of the target loss function, the higher the degree of overlap (i.e., better geometric consistency) of the same physical point in two adjacent viewpoints after backprojecting the matching points onto the bird's-eye view plane according to the current calibration parameters.

[0163] Optionally, based on the current residual of each visual matching point pair obtained in the previous step and the preset robust loss function rule, the robust loss value corresponding to all visual matching point pairs is calculated, and these robust loss values ​​are summed to obtain the current function value of the target loss function. The current function value of the target loss function can reflect the overall inconsistency of all matching point pairs under the current calibration parameters.

[0164] For example, the value of the target loss function is the sum of the squares of the robust loss values ​​for each visual matching point pair. This can be represented as Σ. k ([ ]²), where k is the index of the kth visual matching point pair. The current residual for the k-th visual matching point pair. Let be the robust loss value for the k-th visual matching point pair.

[0165] For the k-th visual matching point pair in the set of visual matching point pairs, when the absolute value of the residual corresponding to the k-th visual matching point pair is less than or equal to a preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the square of the residual corresponding to the k-th visual matching point pair. When the absolute value of the residual corresponding to the k-th visual matching point pair is greater than the preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the product of α, the absolute value of the residual corresponding to the k-th visual matching point pair, and the preset threshold, minus the product of β and the preset threshold.

[0166] Optionally, β can be a constant used to ensure that the robust loss value corresponding to the k-th visual matching point pair remains continuous when the absolute value of the residual corresponding to the k-th visual matching point pair is equal to a preset threshold.

[0167] For example, if |r k If |≤δ, then the visual matching point pair is considered a high-quality visual matching point pair (i.e., inlier), and its corresponding robust loss value ρ(r) k )=r k 2 (i.e., standard squared loss). If |r k|>δ, then this visual matching point pair may contain a false match or noise, and its corresponding robust loss value ρ(r k )=α|r k | δ-β δ.

[0168] In an optional embodiment, the parameters to be optimized for each visual sensor (such as translation vector and rotation matrix parameters) are spliced into a high-dimensional parameter vector x. Assuming there are a total of M visual sensors, and each visual sensor optimizes D parameters, then x∈R M×D . In the current round, x current is the current calibration parameter, the residuals r of all N matching point pairs k are arranged into a column vector, the partial derivative of each residual r k with respect to each optimized parameter x j is calculated to form an N×(M×D) matrix J, the first-order Taylor expansion is used to approximate the objective function, and a system of linear equations for solving the parameter increment Δx is constructed, which can yield the equation of the following formula (14):

[0169]

[0170] wherein, is the transpose of the above matrix, W is the robust weight matrix (which is a diagonal matrix), and the element w_k in W is , , r is the residual vector. According to the calculated calibration parameter increment Δx, the current calibration parameter is updated by the following formula (15):

[0171] x new =x current +Δx (15)

[0172] Using the updated calibration parameter x new , the residual calculation and the loss function calculation are re-executed to obtain the function value L_new of the new objective loss function. If L_new < L_current (the current function value), it indicates that the calibration parameter update is valid. The new calibration parameter is accepted, let x current =x new , and the process proceeds to the next iteration. If L_new ≥ L_current (indicating that the loss has not decreased), it may be necessary to reduce the step size (for example, multiply Δx by a factor less than 1) or adjust the algorithm parameters (for example, increase the damping factor λ in the LM algorithm), and retry the update until the loss decreases or the maximum number of attempts is reached. The above process is repeated until any of the following conditions is met, and the final x current is output as the target calibration parameter: the variation of the objective loss function is less than a preset minimum value; the increment of the calibration parameter is less than a preset threshold (Δx < ); reach the preset maximum number of iterations.

[0173] In some embodiments, a 3D point (in the vehicle coordinate system) is known. It is transformed to the camera coordinate system (i.e., the sensor coordinate system) using the camera's extrinsic parameters (rotation R, translation t). Then, it is projected onto the 2D pixel coordinates of the BEV image using intrinsic parameters (focal length f, principal point c, etc.) and a distortion model (e.g., k1, k2, k3, k4 of the fisheye model), resulting in a 2D point (worldx, worldy) on the BEV image. Now, it is necessary to find the corresponding pixel coordinates (u, v) of this 2D point on the original fisheye image to begin subsequent iterations. Assuming the height coordinate Z = 0 (worldx, worldy, 0) of the 2D point (worldx, worldy) on the BEV image, the 3D ground point is transformed from the vehicle coordinate system to the corresponding sensor coordinate system using the extrinsic parameter matrix of the vision sensor. Finally, the distorted normalized coordinates are transformed to the final image pixel coordinates (u, v) using the intrinsic parameter matrix, thus obtaining the visual matching point pair of the original fisheye image (i.e., the first visual image). The extrinsic parameters (translation and rotation) of multiple visual sensors (such as four cameras (rear, left, front, and right)) are defined as variables to be optimized. Here, the Z-height is fixed and not optimized (because optimizing Z would introduce scale issues). Therefore, a total of 5 parameters are optimized. With 4 = 20 variables, the algorithm's input includes: initial values ​​for the 20 variables, and visual matching point pairs from the original fisheye image (i.e., the first visual image). The optimization process essentially involves finding a point in a 20-dimensional high-dimensional parameter space that minimizes the objective function value (i.e., the sum of squares of all residuals). The objective function is defined as the robust sum of squares of the reprojection residuals of all matching point pairs, and its mathematical expression is as follows: Let the 20-dimensional parameter vector to be optimized be x ∈ R². 0 x contains the translation (X, Y) and rotation (Roll, Pitch, Yaw) parameters of four visual sensors (such as a camera). For each successful image matching point (i.e., each visual matching point pair), it is defined as an observation k. Assuming there are N pairs of visual matching point pairs, and these N pairs of visual matching point pairs are distributed across the overlapping areas of all visual sensors (e.g., front-left, back-left, back-right, front-right), for the k-th observation, its residual r k The calculation method for (x) is as follows: using the intrinsic and extrinsic parameters of the two cameras corresponding to the current parameter x, the pair of image matching points are back-projected onto the assumed ground plane (Z=0) to obtain two three-dimensional world coordinate points P. 1k and P 2k Ignore the Z component (which should theoretically be 0), and calculate them (i.e., P). 1k and P 2kEuclidean distance in the XY plane: r k (x) = || P 1k {xy} - P 2k {xy} ||, this residual r k The physical meaning of (x) is quite intuitive. The residual measures the degree of inconsistency between the backprojected positions of the same physical point in observations from two different camera perspectives under the current calibration parameter estimates. Ideally, this distance should be zero. We can directly minimize the sum of squares of all residuals, i.e., Σ. k [r k The equation [x]² is a standard least squares problem. However, this is very dangerous in practical applications because, despite rigorous matching point filtering (such as RANSAC, slope distance filtering), there may still be unidentified mismatches (outsides) in the data. These outsides generate huge residuals, and due to the presence of the squared term, they have a disproportionately large impact on the optimization objective, thus pulling the optimization process in the wrong direction and causing severe distortion in parameter estimation. To solve this problem, a robust loss function ρ(z) is introduced to "smooth" the squared residuals. The final objective function becomes: F(x) = Σ k ρ( [r k (x)]² ), the Huber loss function is selected as ρ(z), and its definition is as follows (16):

[0174]

[0175] Where z represents the square of the residual [r(x)]², δ is a preset threshold parameter, and the Huber loss function can achieve the following: when the absolute value of the residual |r k When | is small (≤δ), it degenerates into the standard squared loss (ρ(z) = z), which makes the optimization highly efficient when the data fits well, and when the absolute value of the residuals |r k |larger (|r) k When |>δ), it transforms the loss function from a quadratic form to a linear form (ρ(z) ∝ |r) k This significantly weakens the influence of outliers on the overall objective function, as linear growth is much more moderate than quadratic growth. After iterative optimization, the final 20 optimized parameter values ​​were obtained. Projecting the original image onto the BEV view using these new parameters for stitching yielded a satisfactory result. Figure 6 This is a schematic diagram of an optional pre-optimized bird's-eye view stitching effect according to an embodiment of this application. Figure 7 This is a schematic diagram of an optional optimized bird's-eye view stitching effect according to an embodiment of this application. Figure 6 and Figure 7 It can be seen that, Figure 7 The resulting bird's-eye view image of the target is closer to the actual environment around the vehicle. This can correct the initial calibration error and is not limited by the environment, thus achieving better calibration results.

[0176] This embodiment introduces a robust loss function based on the absolute value of residuals, effectively suppressing the impact of mismatched points on calibration results and ensuring the accuracy of calibration parameters. Through multiple rounds of iterative correction, utilizing geometric consistency constraints, it can automatically detect and correct extrinsic parameter drift caused by vehicle bumps, vibrations, or installation errors, without manual intervention or dedicated calibration targets. The robust loss function uses squared loss when the error is small, ensuring high convergence efficiency when the data matches well; when the error is large, it uses linear or hybrid forms to avoid gradient explosion or optimization divergence, enabling the algorithm to converge stably in complex real-world driving scenarios. By correcting the calibration parameters of each visual sensor, it ensures the geometric alignment accuracy of multi-view images during bird's-eye view stitching, eliminating image ghosting and misalignment, and improving the visual perception effect of the surround-view system.

[0177] Optionally, Figure 8 This is a flowchart of an optional bird's-eye view generation according to an embodiment of this application. First, a BEV image is acquired, then BEV matching points are acquired, then the BEV matching points are filtered and selected to acquire the matching points of the original fisheye image (i.e., the first visual image), and the extrinsic parameters in the initial calibration parameters are iteratively optimized.

[0178] Based on the above embodiments, and addressing the shortcomings of related technologies, a self-calibration method for surround-view cameras based on the consistency of matching points in a consensus region is proposed. This method establishes a consensus region within the overlapping area of ​​multiple cameras and utilizes the geometric consistency constraints of the matching point set within the region to construct a robust optimization objective function. This achieves high-precision, real-time self-calibration of the extrinsic parameters of multiple cameras, offering the following significant advantages: It does not rely on specific structural features and has strong scene adaptability; that is, it does not depend on specific geometric structural features such as lane lines, nor does it require rich texture point features. Through the consensus region filtering mechanism, it can still stably extract and optimize high-quality matching regions even in natural scenes with sparse textures, drastic lighting changes, shadow interference, or missing ground markings. This allows the algorithm to maintain high robustness and applicability in complex scenes such as parking lots, tunnels, and rural roads. It also has low computational cost and supports online real-time optimization. By introducing region consistency constraints, a large number of sparse point-level matchings are converted into region-level statistical matchings, significantly reducing the number of variables involved in the optimization. The optimization process relies only on a small number of high-confidence matching regions, resulting in significantly lower computational complexity than traditional texture point methods. It can be executed in real-time in a lightweight manner during vehicle movement, enabling dynamic correction of lens extrinsic parameters and effectively addressing calibration parameter drift caused by lens mounting attitude shifts or vibrations over time. The matching point filtering mechanism effectively avoids interference from non-ground points, ensuring stable convergence during the optimization process. When establishing the consensus region, it combines the surround-view system imaging model and ground constraint assumptions, retaining only matching regions that satisfy ground projection consistency. This avoids the influence of non-ground points (such as roadside objects, vehicles, and walls) on the calibration results from the outset. This mechanism guarantees the physical rationality of the optimization residuals, making the loss function more geometrically consistent, thus significantly improving the stability and convergence speed of the optimization. It exhibits higher robustness under photometric inconsistencies, as the core constraints are based on geometric consistency rather than photometric consistency. Therefore, it is naturally robust to differences in exposure between lenses, differences in lens characteristics, and changes in ambient lighting. Even with strong brightness or color mismatches, it does not significantly affect the accuracy and stability of the calibration results. In summary, the surround-view lens self-calibration method based on consensus region matching point consistency proposed in the above embodiments can achieve high-precision, robust, and real-time online extrinsic parameter optimization without relying on specific calibration objects and ideal environmental conditions. It effectively overcomes the problems of strong feature dependence, unstable matching, and easy divergence in optimization of existing online self-calibration methods.

[0179] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0181] According to another aspect of the embodiments of this application, a vehicle bird's-eye view generation apparatus is also provided. This apparatus can be used to implement the vehicle bird's-eye view generation method provided in the above embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0182] Figure 9 This is a structural block diagram of an optional vehicle bird's-eye view generation apparatus according to an embodiment of this application, such as... Figure 9 As shown, the apparatus for generating the bird's-eye view of the vehicle includes:

[0183] The conversion unit 902 is used to acquire the first visual image of each of the multiple visual sensors on the vehicle, and convert the first visual image of each visual sensor into a bird's-eye view image based on the initial calibration parameters of each visual sensor, thereby obtaining multiple first bird's-eye view images; each pair of adjacent visual sensors in the multiple visual sensors has an overlapping area in the bird's-eye view.

[0184] The matching unit 904 is used to extract feature points from the overlapping area of ​​each first bird's-eye view image and its adjacent first bird's-eye view image in the multiple first bird's-eye view images, and to match the feature points of each pair of adjacent bird's-eye view images in the multiple first bird's-eye view images to obtain multiple pairs of bird's-eye view matching points corresponding to each pair of visual sensors; each pair of bird's-eye view images corresponds to a pair of visual sensors.

[0185] The calibration unit 906 is used to backmap each pair of bird's-eye view matching points corresponding to each pair of vision sensors back to the first vision image of each pair of vision sensors, to obtain multiple pairs of vision matching points corresponding to each pair of vision sensors, and to calibrate the calibration parameters of each pair of vision sensors based on the multiple pairs of vision matching points corresponding to each pair of vision sensors, so as to obtain the target calibration parameters of each vision sensor.

[0186] The execution unit 908 is used to acquire the second visual image of each visual sensor, map the second visual image of each visual sensor to a bird's-eye view image based on the target calibration parameters of each visual sensor, obtain multiple second bird's-eye view images, and stitch the multiple second bird's-eye view images into a target bird's-eye view image.

[0187] It should be noted that the conversion unit 902 in this embodiment can be used to perform the above step S202, the matching unit 904 in this embodiment can be used to perform the above step S204, the correction unit 906 in this embodiment can be used to perform the above step S206, and the execution unit 908 in this embodiment can be used to perform the above step S208.

[0188] The embodiments provided in this application acquire a first visual image from each of the multiple visual sensors on a vehicle. Based on the initial calibration parameters of each visual sensor, the first visual image of each visual sensor is converted into a bird's-eye view image, resulting in multiple first bird's-eye view images. This causes adjacent visual sensors to form overlapping regions under the bird's-eye view perspective. Feature points are then extracted and matched from the overlapping regions of adjacent first bird's-eye view images to obtain multiple corresponding bird's-eye view matching point pairs. Each bird's-eye view matching point pair is then back-mapped back to the corresponding first visual image, resulting in multiple visual matching point pairs. Finally, the initial calibration parameters of each visual sensor are corrected based on these multiple visual matching point pairs to obtain target calibration parameters. By utilizing the consistency constraints of multiple pairs of visual matching points in the overlapping area under multiple perspectives, geometric relationships can be established in the original image space through an inverse mapping mechanism. This directly optimizes the calibration parameters to eliminate reprojection errors, thereby correcting the deviations caused by the initial calibration. Finally, based on the corrected target calibration parameters, the second visual image of the visual sensor is mapped to a second bird's-eye view image and stitched together to form the target bird's-eye view image. This allows the generated bird's-eye view to accurately reflect the real geometric relationships of the vehicle's surrounding environment, improving the accuracy of the bird's-eye view. This can solve the technical problem in related technologies where the use of fixed calibration parameters during image generation leads to significant errors between the generated bird's-eye view and the actual scene of the vehicle, resulting in inaccurate bird's-eye views.

[0189] In an exemplary embodiment, the conversion unit is configured to perform the following conversion operations on each visual sensor as the current visual sensor, wherein the initial calibration parameters of the current visual sensor are current initial calibration parameters, and the first visual image of the current visual sensor is the current first visual image: based on a first mapping relationship, the bird's-eye view pixel coordinates of each bird's-eye view pixel in the current bird's-eye view image canvas are mapped to world coordinates in the world coordinate system, to obtain the world coordinates corresponding to each bird's-eye view pixel, wherein the current bird's-eye view image canvas is a digital image created for the current first visual image, and the first mapping relationship is the mapping relationship between the pixel coordinates of the bird's-eye view pixel in the current bird's-eye view image canvas and the world coordinates in the world coordinate system; based on the current initial calibration parameters, the world coordinates corresponding to each bird's-eye view pixel are converted to pixel coordinates in the pixel coordinate system of the current first visual image, to obtain the original image pixel coordinates corresponding to each bird's-eye view pixel; the pixel value of the original image pixel coordinates corresponding to each bird's-eye view pixel is assigned to each bird's-eye view pixel in the current bird's-eye view image canvas, to obtain the bird's-eye view image corresponding to the current first visual image.

[0190] In an exemplary embodiment, the world coordinate system is the vehicle coordinate system of the vehicle, and the origin of the world coordinate system is a designated point on the vehicle; the transformation unit is configured to obtain a preset bird's-eye view area and a preset image resolution before mapping the bird's-eye pixel coordinates of each bird's-eye view pixel in the current bird's-eye view image canvas to world coordinates in the world coordinate system based on a first mapping relationship, wherein the preset bird's-eye view area is a region within the bird's-eye plane of the world coordinate system set for the bird's-eye view image to be converted from the current first visual image, and the preset image resolution is an image resolution set for the bird's-eye view image to be converted from the current first visual image; and determine the first mapping relationship based on the region size of the preset bird's-eye view area and the preset image resolution.

[0191] In an exemplary embodiment, the current initial calibration parameters include current initial calibration extrinsic parameters; a conversion unit is used to convert the world coordinates corresponding to each bird's-eye view pixel into local coordinates in the sensor coordinate system of the current visual sensor based on the current initial calibration extrinsic parameters, to obtain the sensor local coordinates corresponding to each bird's-eye view pixel; and using the calibration intrinsic parameters of the current visual sensor and the sensor distortion model of the current visual sensor, converting the sensor local coordinates corresponding to each bird's-eye view pixel into pixel coordinates in the pixel coordinate system of the current first visual image, to obtain the original image pixel coordinates corresponding to each bird's-eye view pixel.

[0192] In an exemplary embodiment, a feature point of each first bird's-eye view image is represented by a feature point position and a descriptor vector. The matching unit is configured to perform the following feature point matching operation on each pair of bird's-eye view images as the current pair of bird's-eye view images, obtaining multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors. The current pair of bird's-eye view images includes a first current bird's-eye view image and a second current bird's-eye view image: embedding encoding is performed on the descriptor vector of each feature point in the first current bird's-eye view image, and position encoding is performed on the feature point position of each feature point in the first current bird's-eye view image to obtain a first current feature sequence corresponding to the first current bird's-eye view image. The pair of visual sensors corresponding to the current pair of bird's-eye view images is the current pair of visual sensors. Embedding encoding is performed on the descriptor vector of each feature point in the second current bird's-eye view image. The location of each feature point in the second current bird's-eye view image is encoded to obtain the second current feature sequence corresponding to the second current bird's-eye view image. The first and second current feature sequences are input into an M-layer decoder, and the set of multiple feature point pairs indicated by the matching results output by each layer decoder is determined as a candidate matching point pair set. Each layer decoder is used to perform cross-attention matching on the two input feature sequences and outputs indication information of feature point pairs with a matching confidence greater than or equal to a preset confidence threshold. M is a positive integer greater than or equal to 1. Multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors are selected from the candidate matching point pair set. The number of bird's-eye view matching point pairs in the multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors is less than or equal to a preset number.

[0193] In an exemplary embodiment, the matching unit is configured to: select all candidate matching point pairs in the candidate matching point pair set as multiple bird's-eye view matching point pairs corresponding to the current visual sensor when the number of candidate matching point pairs in the candidate matching point pair set is less than or equal to a preset number; perform consistency verification on each candidate matching point pair in the candidate matching point pair set when the number of candidate matching point pairs in the candidate matching point pair set is greater than the preset number; remove candidate matching point pairs that fail the consistency verification in the candidate matching point pair set to obtain an updated candidate matching point pair set; and select all candidate matching point pairs in the updated candidate matching point pair set as multiple bird's-eye view matching point pairs corresponding to the current visual sensor.

[0194] In one exemplary embodiment, the matching unit is configured to: determine the reprojection error between two candidate matching points in each candidate matching point pair, thereby obtaining the reprojection error corresponding to each candidate matching point pair; perform a consistency check on each candidate matching point pair based on the reprojection error corresponding to each candidate matching point pair; determine the slope direction of the line connecting the two candidate matching points in each candidate matching point pair, thereby obtaining the slope direction corresponding to each candidate matching point pair; perform a consistency check on each candidate matching point pair based on the slope direction corresponding to each candidate matching point pair; determine the Euclidean distance between two candidate matching points in each candidate matching point pair, thereby obtaining the Euclidean distance corresponding to each candidate matching point pair; and perform a consistency check on each candidate matching point pair based on the Euclidean distance corresponding to each candidate matching point pair.

[0195] In an exemplary embodiment, the calibration unit is configured to perform multiple rounds of correction operations on the initial calibration parameters of each visual sensor to obtain the target calibration parameters of each visual sensor. During the current round of correction operations, the initial calibration parameters of each visual sensor before correction are the current calibration parameters of each visual sensor. The set of multiple visual matching point pairs corresponding to each pair of visual sensors is called the visual matching point pair set. Based on the current calibration parameters of each visual sensor, the Euclidean distance of each visual matching point pair in the visual matching point pair set on the bird's-eye view plane is determined to obtain the current residual corresponding to each visual matching point pair. Based on the current residual of each visual matching point pair, the current function value of the target loss function is determined. Based on the current residual of the target loss function... The function value is used to correct the current calibration parameters of each vision sensor to reduce the value of the target loss function. The value of the target loss function is the sum of the robust loss values ​​corresponding to each visual matching point pair. For the k-th visual matching point pair in the set of visual matching point pairs, when the absolute value of the residual corresponding to the k-th visual matching point pair is less than or equal to a preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the square of the residual corresponding to the k-th visual matching point pair. When the absolute value of the residual corresponding to the k-th visual matching point pair is greater than the preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the product of α, the absolute value of the residual corresponding to the k-th visual matching point pair and the preset threshold, minus the product of β and the preset threshold, where α≥1 and β>0.

[0196] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0197] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0198] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0199] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0200] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0201] According to another aspect of the embodiments of this application, a computer program product is also provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit 1001, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0202] Figure 10 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 10 As shown, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in ROM 1002 or programs loaded into RAM 1003 from storage section 1008. Random access memory 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0203] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a local area network card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. Drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0204] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit 1001, it performs various functions defined in the system of this application.

[0205] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0206] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0207] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating a bird's-eye view of a vehicle, characterized in that, include: The first visual image of each of the multiple visual sensors on the vehicle is acquired, and based on the initial calibration parameters of each visual sensor, the first visual image of each visual sensor is converted into a bird's-eye view image to obtain multiple first bird's-eye view images; each pair of adjacent visual sensors in the multiple visual sensors has an overlapping area in the bird's-eye view. Feature points are extracted from the overlapping areas of each first bird's-eye view image and its adjacent first bird's-eye view images in the plurality of first bird's-eye view images, and feature points of each pair of adjacent bird's-eye view images in the plurality of first bird's-eye view images are matched to obtain multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors; each pair of bird's-eye view images corresponds to a pair of visual sensors. Each pair of bird's-eye view matching points corresponding to each pair of visual sensors is back-mapped back to the first visual image of each pair of visual sensors to obtain multiple visual matching point pairs corresponding to each pair of visual sensors. Based on the multiple visual matching point pairs corresponding to each pair of visual sensors, the initial calibration parameters of each visual sensor are corrected to obtain the target calibration parameters of each visual sensor. The second visual image of each visual sensor is acquired. Based on the target calibration parameters of each visual sensor, the second visual image of each visual sensor is mapped to a bird's-eye view image to obtain multiple second bird's-eye view images. The multiple second bird's-eye view images are then stitched together to form a target bird's-eye view image.

2. The method according to claim 1, characterized in that, Based on the initial calibration parameters of each visual sensor, the first visual image of each visual sensor is converted into a bird's-eye view image, resulting in multiple first bird's-eye view images, including: Perform the following conversion operation on each of the aforementioned vision sensors as the current vision sensor, wherein the initial calibration parameters of the current vision sensor are the current initial calibration parameters, and the first visual image of the current vision sensor is the current first visual image: Based on the first mapping relationship, the bird's-eye view pixel coordinates of each bird's-eye view pixel in the current bird's-eye view image canvas are mapped to world coordinates in the world coordinate system to obtain the world coordinates corresponding to each bird's-eye view pixel. The current bird's-eye view image canvas is a digital image created for the current first visual image, and the first mapping relationship is the mapping relationship between the pixel coordinates of the bird's-eye view pixels in the current bird's-eye view image canvas and the world coordinates in the world coordinate system. Based on the current initial calibration parameters, the world coordinates corresponding to each bird's-eye view pixel are converted into pixel coordinates in the pixel coordinate system of the current first visual image, so as to obtain the original image pixel coordinates corresponding to each bird's-eye view pixel. Assign the pixel value of the original image pixel coordinates corresponding to each bird's-eye view pixel to each bird's-eye view pixel in the current bird's-eye view image canvas to obtain the bird's-eye view image corresponding to the current first visual image.

3. The method according to claim 2, characterized in that, The world coordinate system is the vehicle coordinate system of the vehicle, and the origin of the world coordinate system is a designated point on the vehicle. Before mapping the bird's-eye view pixel coordinates of each bird's-eye view pixel in the current bird's-eye view image canvas to world coordinates in the world coordinate system based on the first mapping relationship, the method further includes: Obtain a preset bird's-eye view area and a preset image resolution, wherein the preset bird's-eye view area is a region within the bird's-eye view plane of the world coordinate system set for the bird's-eye view image to be converted from the current first visual image, and the preset image resolution is an image resolution set for the bird's-eye view image to be converted from the current first visual image; The first mapping relationship is determined based on the area size of the preset bird's-eye view region and the preset image resolution.

4. The method according to claim 2, characterized in that, The current initial calibration parameters include the current initial calibration extrinsic parameters; The step of converting the world coordinates corresponding to each bird's-eye view pixel into pixel coordinates in the pixel coordinate system of the current first visual image based on the current initial calibration parameters, to obtain the original image pixel coordinates corresponding to each bird's-eye view pixel, includes: Based on the current initial calibration extrinsic parameters, the world coordinates corresponding to each bird's-eye view pixel are converted into local coordinates in the sensor coordinate system of the current visual sensor, thus obtaining the sensor local coordinates corresponding to each bird's-eye view pixel. Using the calibration intrinsic parameters of the current visual sensor and the sensor distortion model of the current visual sensor, the sensor local coordinates corresponding to each bird's-eye view pixel are converted into pixel coordinates in the pixel coordinate system of the current first visual image, thus obtaining the original image pixel coordinates corresponding to each bird's-eye view pixel.

5. The method according to claim 1, characterized in that, Each feature point in the first bird's-eye view image is represented by a feature point position and a descriptor vector; The step of matching feature points of each pair of adjacent bird's-eye view images in the plurality of first bird's-eye view images to obtain multiple pairs of bird's-eye view matching points corresponding to each pair of visual sensors includes: Each pair of bird's-eye view images is treated as the current pair of bird's-eye view images, and the following feature point matching operation is performed to obtain multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors. The current pair of bird's-eye view images includes a first current bird's-eye view image and a second current bird's-eye view image. The descriptor vector of each feature point in the first current bird's-eye view image is embedded and encoded, and the feature point position of each feature point in the first current bird's-eye view image is position encoded to obtain the first current feature sequence corresponding to the first current bird's-eye view image. The pair of visual sensors corresponding to the current pair of bird's-eye view images is the current pair of visual sensors. The descriptor vector of each feature point in the second current bird's-eye view image is embedded and encoded, and the feature point position of each feature point in the second current bird's-eye view image is position encoded to obtain the second current feature sequence corresponding to the second current bird's-eye view image; The first current feature sequence and the second current feature sequence are input into the M-layer decoder, and the set of multiple feature point pairs indicated by the matching results output by each layer decoder is determined as the candidate matching point pair set. Each layer decoder is used to perform cross-attention matching on the two input feature sequences and output indication information of feature point pairs with matching confidence greater than or equal to a preset confidence threshold. M is a positive integer greater than or equal to 1. From the candidate matching point pair set, select multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors, wherein the number of bird's-eye view matching point pairs in the multiple bird's-eye view matching point pairs corresponding to the current pair of visual sensors is less than or equal to a preset number.

6. The method according to claim 5, characterized in that, The step of selecting multiple bird's-eye view matching point pairs corresponding to the current visual sensor from the candidate matching point pair set includes: If the number of candidate matching point pairs in the candidate matching point pair set is less than or equal to the preset number, all candidate matching point pairs in the candidate matching point pair set will be selected as the multiple bird's-eye view matching point pairs corresponding to the current visual sensor. If the number of candidate matching point pairs in the candidate matching point pair set is greater than the preset number, a consistency check is performed on each candidate matching point pair in the candidate matching point pair set; candidate matching point pairs that fail the consistency check are removed from the candidate matching point pair set to obtain an updated candidate matching point pair set; all candidate matching point pairs in the updated candidate matching point pair set are selected as the multiple bird's-eye view matching point pairs corresponding to the current visual sensor.

7. The method according to claim 6, characterized in that, The consistency check performed on each candidate matching point pair in the candidate matching point pair set includes at least one of the following: Determine the reprojection error between the two candidate matching points in each candidate matching point pair to obtain the reprojection error corresponding to each candidate matching point pair; based on the reprojection error corresponding to each candidate matching point pair, perform consistency verification on each candidate matching point pair respectively; Determine the slope direction of the line connecting the two candidate matching points in each candidate matching point pair to obtain the slope direction corresponding to each candidate matching point pair; Based on the slope direction corresponding to each candidate matching point pair, a consistency check is performed on each candidate matching point pair. Determine the Euclidean distance between the two candidate matching points in each candidate matching point pair to obtain the Euclidean distance corresponding to each candidate matching point pair; Based on the Euclidean distance corresponding to each candidate matching point pair, a consistency check is performed on each candidate matching point pair.

8. The method according to any one of claims 1 to 7, characterized in that, The step of correcting the initial calibration parameters of each visual sensor based on multiple visual matching point pairs corresponding to each pair of visual sensors to obtain the target calibration parameters of each visual sensor includes: Multiple rounds of correction operations are performed on the calibration parameters of each vision sensor to obtain the target calibration parameters of each vision sensor. During the current round of correction operations, the initial calibration parameters of each vision sensor before correction are the current calibration parameters of each vision sensor. The set of multiple visual matching point pairs corresponding to each pair of vision sensors is the visual matching point pair set. Based on the current calibration parameters of each visual sensor, determine the Euclidean distance of each visual matching point pair in the set of visual matching point pairs on the bird's-eye view plane, and obtain the current residual corresponding to each visual matching point pair; Based on the current residual of each visual matching point pair, determine the current function value of the target loss function; Based on the current function value of the target loss function, the current calibration parameters of each visual sensor are corrected to reduce the function value of the target loss function; Wherein, the function value of the target loss function is the sum of the robust loss values ​​corresponding to each visual matching point pair; for the k-th visual matching point pair in the set of visual matching point pairs, when the absolute value of the residual corresponding to the k-th visual matching point pair is less than or equal to a preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the square of the residual corresponding to the k-th visual matching point pair; when the absolute value of the residual corresponding to the k-th visual matching point pair is greater than the preset threshold, the robust loss value corresponding to the k-th visual matching point pair is the product of α, the absolute value of the residual corresponding to the k-th visual matching point pair and the preset threshold, minus the product of β and the preset threshold, where α≥1, β>0.

9. A device for generating a bird's-eye view of a vehicle, characterized in that, include: A conversion unit is used to acquire a first visual image of each of the multiple visual sensors on the vehicle, and convert the first visual image of each visual sensor into a bird's-eye view image based on the initial calibration parameters of each visual sensor, thereby obtaining multiple first bird's-eye view images; each pair of adjacent visual sensors in the multiple visual sensors has an overlapping area in the bird's-eye view. The matching unit is used to extract feature points from the overlapping areas of each first bird's-eye view image and its adjacent first bird's-eye view images in the plurality of first bird's-eye view images, and to match the feature points of each pair of adjacent bird's-eye view images in the plurality of first bird's-eye view images to obtain multiple bird's-eye view matching point pairs corresponding to each pair of visual sensors; each pair of bird's-eye view images corresponds to a pair of visual sensors. The calibration unit is used to backmap each bird's-eye view matching point pair corresponding to each pair of visual sensors back to the first visual image of each pair of visual sensors, to obtain multiple visual matching point pairs corresponding to each pair of visual sensors, and to calibrate the initial calibration parameters of each visual sensor based on the multiple visual matching point pairs corresponding to each pair of visual sensors, to obtain the target calibration parameters of each visual sensor. An execution unit is configured to acquire a second visual image from each visual sensor, map the second visual image of each visual sensor to a bird's-eye view image based on the target calibration parameters of each visual sensor, obtain multiple second bird's-eye view images, and stitch the multiple second bird's-eye view images together to form a target bird's-eye view image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images