Image seam line determination method, computing device, and vehicle

Abstract

The embodiment of the application provides a kind of image of seam line determination method, it can be applied to image splicing technical field, comprising, obtaining calibration data, wherein, calibration data can include the real coordinates of calibration reference object feature point in vehicle coordinate system and the pixel coordinates on camera image.Based on camera parameter, pixel coordinates are projected to vehicle coordinate system to obtain projection coordinates, and compared with real coordinates, the re-projection error of each feature point is obtained, finally, in the visual field overlap area of adjacent vehicle-mounted camera, according to these calculated re-projection error values to make decisions, and then determine the target seam line of the minimum re-projection error, image splicing can be carried out using target seam line to reduce the splicing misplacement and ghosting caused by coordinate mapping deviation to the greatest extent, so that the best seam position under corresponding calibration data is found for each vehicle with different installation conditions, which fundamentally overcomes the defect that preset seam line cannot adapt to individual differences.

Description

Technical Field

[0001] This disclosure relates to the field of image stitching technology, and more particularly to a method for determining stitching lines in images, a computing device, and a vehicle. Background Technology

[0002] In vehicle driver assistance systems, panoramic surround view imaging technology can provide drivers with a bird's-eye view of the vehicle's surroundings by stitching together images captured by multiple onboard cameras, greatly improving safety when parking and driving at low speeds.

[0003] One of the core aspects of this technology is determining an optimal stitching line within the overlapping area of ​​the fields of view of adjacent cameras to achieve seamless stitching of adjacent images, avoid ghosting or misalignment, and ensure the accuracy and visual effect of panoramic images.

[0004] Currently, the common solution is to pre-set the same stitching line for each vehicle. However, due to the unavoidable individual differences in the camera installation position and angle of each vehicle, deviations or defects may occur in the image stitching area in some cases. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this disclosure provide a method, apparatus, computing device, and vehicle for determining stitching lines in images.

[0006] A first aspect of this disclosure provides a method for determining stitching lines in an image, the method comprising: The calibration data is acquired, which may include the true coordinates of the feature points of the calibration reference object in the vehicle coordinate system and the pixel coordinates on the camera image. Based on the camera parameters, the pixel coordinates are projected onto the vehicle coordinate system to obtain the projected coordinates, and the reprojection error of each feature point is calculated. Finally, within the overlapping area of ​​the field of view of adjacent vehicle cameras, decisions are made based on these calculated reprojection error values, thereby determining the target stitching line with the smallest reprojection error. Using the target stitching line for image stitching can minimize stitching misalignment and ghosting caused by coordinate mapping deviations, thus finding the optimal stitching position under the corresponding calibration data for each vehicle with different installation conditions, fundamentally overcoming the defect that the preset stitching line cannot adapt to individual differences.

[0007] A second aspect of this disclosure provides an image stitching line determination device, which is installed in a vehicle and includes: The acquisition unit is used to acquire the calibration data of the vehicle-mounted camera in the target vehicle. The calibration data includes the true coordinates of the feature points on the calibration reference in the vehicle coordinate system and the pixel coordinates on the camera image. The first calculation unit is used to calculate the reprojection error of feature points based on calibration data. The reprojection error is used to characterize the deviation between the projected coordinates of the pixel coordinates mapped to the vehicle coordinate system through camera parameters and the real coordinates. The first determining unit is used to determine the target stitching line for image stitching based on the reprojection error of feature points within the overlapping field of view of adjacent vehicle cameras.

[0008] A third aspect of this disclosure provides a computing device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method of the first aspect.

[0009] A fourth aspect of this disclosure provides a vehicle that includes the computing device of the third aspect. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a method for determining stitching lines in an image according to an embodiment of this disclosure; Figure 2 This is a flowchart illustrating another method for determining stitching lines in an image provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of an image stitching line determination device provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a computing device according to an embodiment of this disclosure. Detailed Implementation

[0013] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0014] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0015] In vehicle driver assistance systems, panoramic surround view imaging technology can provide drivers with a bird's-eye view of the vehicle's surroundings by stitching together images captured by multiple onboard cameras, greatly improving safety when parking and driving at low speeds.

[0016] One of the core aspects of this technology is determining an optimal stitching line within the overlapping area of ​​the fields of view of adjacent cameras to achieve seamless stitching of adjacent images, avoid ghosting or misalignment, and ensure the accuracy and visual effect of panoramic images.

[0017] Currently, the common approach is to pre-set the same stitching line for each vehicle. However, due to unavoidable individual differences in the camera installation position and angle for each vehicle, the optimal stitching position also varies. This method is difficult to adapt to the actual installation situation of all vehicles and cannot guarantee finding the truly optimal stitching position for every vehicle, which may lead to deviations or defects in the image stitching area in some cases.

[0018] In view of this, this application provides a method for determining the stitching line of an image. Instead of pre-setting or randomly selecting the stitching position, this application first acquires calibration data, which may include the true coordinates of feature points of a calibration reference object in the vehicle coordinate system and the pixel coordinates on the camera image. Subsequently, based on camera parameters, the projected coordinates of these pixel coordinates are calculated and reprojected into the vehicle coordinate system, and compared with the true coordinates to obtain the reprojection error of each feature point. This reprojection error directly reflects the accuracy difference of the camera model in different regions. Finally, within the overlapping area of ​​the fields of view of adjacent vehicle cameras, a decision is made based on these calculated reprojection error values. Since the magnitude of the reprojection error directly reflects the accuracy of the image mapping, the target stitching line with the smallest reprojection error is determined. Using the target stitching line for image stitching can minimize stitching misalignment and ghosting caused by coordinate mapping deviations, thereby finding the optimal stitching position under the corresponding calibration data for each vehicle with different installation conditions, fundamentally overcoming the defect that pre-set stitching lines cannot adapt to individual differences.

[0019] Figure 1 This is a flowchart illustrating a method for determining stitching lines in an image according to an embodiment of this disclosure. This method can be executed by a computing device. The computing device may include, but is not limited to, devices such as in-vehicle equipment or vehicle controllers, where the vehicle is equipped with multiple in-vehicle cameras.

[0020] like Figure 1As shown, the method provided in this embodiment includes the following steps: S101: Obtain calibration data of the vehicle-mounted camera in the target vehicle.

[0021] The computing device can acquire calibration data from an onboard camera in the target vehicle. The onboard camera is an image capture device installed on the vehicle for taking images. The calibration data can include the true coordinates of feature points on a calibration reference in the vehicle coordinate system and their pixel coordinates in the camera image. The vehicle coordinate system is a three-dimensional spatial system with the vehicle as a reference. The true coordinates are the actual three-dimensional positions of the feature points in this coordinate system. The camera image refers to a two-dimensional image (such as a fisheye image) captured by the onboard camera, and the pixel coordinates are the two-dimensional pixel positions of the feature points in the image. The calibration reference can be, for example, a checkerboard pattern. A checkerboard is a reference used for camera calibration; its regular grid pattern provides feature points with known three-dimensional spatial coordinates. Feature points can be, for example, corner points, which are the intersections of grid lines in the checkerboard pattern. These can be used as feature points for calibration. This is just an example and not a specific limitation; those skilled in the art can set the calibration reference according to actual needs.

[0022] In some possible implementations, the calibration data may also include projected coordinates, which can be determined in the following ways: The computing device acquires a fisheye image of a calibration reference object captured by an onboard camera, determines the pixel coordinates of feature points on the calibration reference object from the fisheye image, calculates the extrinsic parameters of the onboard camera based on the true coordinates of the feature points in the vehicle coordinate system and the corresponding pixel coordinates, and finally transforms the pixel coordinates of the feature points to the projected coordinates in the vehicle coordinate system using the intrinsic parameters, distortion coefficients and extrinsic parameters of the onboard camera.

[0023] For example, feature points on the calibration reference can be represented by the corner points of a checkerboard pattern (e.g., P1, P2, P3...). The positions of these corner points in the fisheye image are their pixel coordinates. Meanwhile, the true coordinates of these corner points in the vehicle coordinate system (i.e., their actual 3D positions on the ground, such as G1, G2, G3...) are known in advance. Using the known correspondence between pixel coordinates and true coordinates, the computing device can utilize the `solvePnP` function, combined with the Ceres library, to perform nonlinear optimization to calculate the extrinsic parameters of the vehicle camera (i.e., the rotation matrix R and translation vector t, which define the relative position and orientation between the camera coordinate system and the vehicle coordinate system). The `solvePnP` function is used to solve the 3D to 2D point-to-point perspective problem; it can calculate the initial extrinsic parameters of the camera based on the true coordinates (in the vehicle coordinate system) and corresponding pixel coordinates of the feature points on the calibration reference. The Ceres library is used to solve large-scale nonlinear optimization problems; in this application, it is used to perform nonlinear optimization on the initial extrinsic parameters obtained by `solvePnP`, minimizing the reprojection error to obtain more accurate camera extrinsic parameter results.

[0024] After completing the extrinsic parameter calculations, the computing device can combine the known intrinsic parameters of the vehicle-mounted camera (e.g., focal lengths fx, fy and principal point coordinates cx, cy) with distortion coefficients (k1, k2, k3, k4, used to correct fisheye image distortion) to construct a complete camera model. Finally, using this camera model, the pixel coordinates of the feature points are projected back into three-dimensional space, and their projected coordinates in the vehicle coordinate system (e.g., H1, H2, H3...) are calculated. The deviation between these projected coordinates and the true coordinates is the reprojection error, which is the fundamental basis for determining the optimal stitching line in this application.

[0025] S102: Calculate the reprojection error of feature points based on calibration data.

[0026] The computing device can calculate the reprojection error of feature points based on calibration data. This reprojection error characterizes the deviation between the projected coordinates and the true coordinates. Specifically, it refers to the deviation between the projected coordinates of a feature point, obtained by reprojecting the pixel coordinates of a feature point onto the vehicle coordinate system using the complete camera model, and its known true coordinates. A larger reprojection error value indicates a less accurate mapping of the camera model at that feature point.

[0027] S103: Within the overlapping field of view of adjacent vehicle cameras, determine the target stitching line for image stitching based on the reprojection error of feature points.

[0028] The computing device can determine the target stitching line for image stitching based on the reprojection error of feature points within the overlapping field of view of adjacent vehicle cameras. The overlapping field of view specifically refers to the area where the independent fields of view of any two adjacent cameras overlap and are jointly covered in a panoramic bird's-eye view model constructed by multiple vehicle cameras (in this embodiment, we will use four cameras: front, rear, left, and right). Figure 2 As shown, for example, the overlapping areas of the field of view can be four regions: A, B, C, and D.

[0029] For example, after the fisheye images captured by each vehicle-mounted camera are corrected and projected onto the ground model in the vehicle coordinate system, each camera's viewpoint forms a specific coverage area. At the edge of the coverage areas of two adjacent vehicle-mounted cameras (e.g., the front-view camera and the left-view camera), there exists a strip-shaped region where the same physical point (such as a checkerboard corner point on the ground) will appear simultaneously in the images of both cameras. This strip-shaped region is the field-of-view overlap area.

[0030] In some possible implementations, for a specific overlapping region of the field of view (which can be called the target overlapping region of the field of view), the target stitching line for image stitching is determined based on the reprojection error of feature points. Specifically, it can be: The computing device can determine the target stitching line based on the reprojection error distribution of feature points within the overlapping region of the target field of view, with the optimization objective of minimizing the total reprojection error within the two sub-regions divided by the candidate stitching line. That is, the final target stitching line is determined based on the distribution of reprojection errors of feature points within the overlapping region of the target field of view, using the criterion of minimizing the total reprojection error. Specifically, for any candidate stitching line, the corresponding total reprojection error is determined based on the reprojection error of feature points located within each of the divided sub-regions.

[0031] In some possible implementations, the seam line can be a straight line. Multiple candidate seam lines can be determined first, and then the target seam line with the smallest total weight projection error can be determined from the multiple candidate seam lines.

[0032] The computing device can determine a reference point within the overlapping area of ​​the target field of view. Starting from the reference point, multiple candidate stitching lines are determined, each corresponding to a direction angle. For each candidate stitching line, the overlapping area of ​​the target field of view is divided into a first sub-region and a second sub-region, with the candidate stitching line as the boundary. Then, the first super-projection error calculated using the camera parameters of the first camera can be determined for feature points in the first sub-region, and the second super-projection error calculated using the camera parameters of the second camera can be determined for feature points in the second sub-region. The corresponding total super-projection error is determined using the first and second projection errors of the same candidate stitching line. Finally, the candidate stitching line with the smallest total super-projection error can be determined as the target stitching line from among the multiple candidate stitching lines.

[0033] For example, the computing device determines a reference point as the starting point within the overlapping area of ​​the target field of view. Taking region A as an example, the reference point is typically chosen as the projected corner point of the lower right corner of the region, denoted as H1 (i.e., the projected coordinates of the feature point in the vehicle coordinate system). Then, starting from H1, multiple candidate seam lines are generated. Each candidate seam line corresponds to a direction angle, defined by the angle between the line connecting H1 and other projected corner points Hn within the region and the horizontal line to the left of H1. The angle range can be limited, for example, between 20° and 70° to ensure that the seam line is located within a reasonable overlapping area of ​​the field of view. For example, all corner points Hn within region A are traversed, and only when the angle meets this range is the connecting line considered a candidate seam line for subsequent processing. For each candidate seam line, the computing device can divide the overlapping area of ​​the target field of view into a first sub-region and a second sub-region, using the seam line as a boundary. In region A, the first sub-region corresponds to the area below the seam line, and the second sub-region corresponds to the area above the seam line. Then, the first reprojection error, calculated using the camera parameters of the first camera (e.g., the left-view camera), and the second reprojection error, calculated using the camera parameters of the second camera (e.g., the front-view camera), are calculated for feature points in the first sub-region. The computing device then sums the first and second reprojection errors corresponding to the same candidate seam line to obtain the total reprojection error. This process aims to minimize the total reprojection error, as a larger reprojection error indicates lower mapping accuracy of the camera model in that region and a less than ideal seam line position. Finally, the seam line with the smallest total reprojection error is selected from all candidate seam lines and identified as the target seam line.

[0034] This method ensures that each vehicle obtains a customized seam position due to differences in camera installation, avoiding the limitations of preset seams. Furthermore, through the above steps, this application can complete seam line optimization offline, improving image stitching quality and avoiding visual abrupt changes caused by dynamic adjustments during runtime.

[0035] In some possible implementations, the computing device can use the projected coordinates of each feature point as path nodes within the overlapping region of the target field of view to determine the start point, end point, and connectivity rules between path nodes. Starting from the start point, feasible paths leading to the end point are traversed according to the connectivity rules. Then, for each feasible path, the overlapping region of the target field of view is divided into a first sub-region and a second sub-region, with the feasible path as a candidate stitching line. The first super-projection error calculated using the camera parameters of the first camera is determined for the feature points in the first sub-region, and the second super-projection error calculated using the camera parameters of the second camera is determined for the feature points in the second sub-region. Finally, the corresponding total super-projection error can be determined using the first and second projection errors of the same feasible path. From all feasible paths, the feasible path with the smallest total super-projection error is determined as the target stitching line.

[0036] For example, in the target field of view overlap area (with Figure 2 In region A of the map, the projected coordinates (such as H1, H2, H3, ...) of each feature point are projected onto the vehicle coordinate system using camera parameters and used as path nodes.

[0037] The computing device can determine the start and end points of the path. For example, a projected corner point in the lower right corner of the region can be used as the path start point, denoted as H1, and a virtual point F in the upper left corner of the region can be used as the path end point. Simultaneously, connectivity rules between path nodes can be defined. In this embodiment, the connectivity rules can be limited to allowing movement only from the current node to the left or upward to connect to the next adjacent corner point Hn.

[0038] Starting from the origin H1, the computing device can traverse all feasible paths that can reach the destination F according to the connectivity rules described above. When a path cannot be continued from a certain node, it can be directly connected to the destination F. Each complete and continuous path from the origin to the destination can be considered as a candidate seam line.

[0039] For each feasible path obtained through traversal, the computing device can use it as a candidate seam line to divide the overlapping area of ​​the target field of view into two sub-regions: a first sub-region (e.g., the region to the left of the candidate seam line) and a second sub-region (e.g., the region to the right of the candidate seam line). Then, the first reprojection error, calculated using the camera parameters of a first camera (e.g., a left-view camera), is calculated for feature points within the first sub-region, and the second reprojection error, calculated using the camera parameters of a second camera (e.g., a front-view camera), is calculated for feature points within the second sub-region. The first and second reprojection errors corresponding to the same feasible path are added together to obtain the total reprojection error for that path.

[0040] Ultimately, the computing device can select the path that minimizes the total weight projection error from all the feasible paths traversed, and determine it as the final target seam line.

[0041] While this stepped polyline scheme is more complex than simple ray angle traversal, its advantage lies in its ability to traverse all complex paths formed by corner points. This allows for more flexible adaptation to the distribution of feature point reprojection errors, potentially leading to better stitching shapes than fixed-angle rays. This further improves the stitching quality of panoramic images.

[0042] In some possible implementations, the computing device can detect the change in the vehicle suspension height of the target vehicle relative to the calibrated default height. If the absolute value of the change is greater than or equal to a preset change threshold, the extrinsic parameters of the on-board camera can be updated according to the vehicle suspension height to obtain the corrected extrinsic parameters. Based on the corrected extrinsic parameters, the projected coordinates of the feature points in the vehicle coordinate system are calculated. Using the recalculated projected coordinates, the corrected reprojection error of the feature points is calculated. Finally, the target stitching line for image stitching can be determined within the overlapping area of ​​the field of view based on the corrected reprojection error.

[0043] In some possible implementations, to ensure the accuracy of panoramic image stitching under different vehicle conditions, the computing device can dynamically correct the stitching seam position to cope with the impact of changes in vehicle suspension height, enabling the stitching lines to adapt to the real-time vehicle posture.

[0044] For example, when the vehicle is running, the computing device can detect the vehicle's suspension height and compare it with the calibrated default height. If the absolute value of the change is greater than or equal to a preset change threshold, it indicates that the suspension height deviation may significantly impair the accuracy of the camera's extrinsic parameters, and the computing device can then initiate a correction process.

[0045] The correction process can begin by updating the extrinsic parameters of the onboard cameras based on the vehicle's suspension height. If the suspension plane is parallel to the horizontal plane, the translation vector height value in the extrinsic parameters can be directly adjusted, i.e., the change is added to the translation vectors of the four cameras; If the suspension plane angle changes, the camera's rotation matrix R needs to be multiplied by the angle change R' to obtain the corrected extrinsic parameters. This step ensures that the mapping relationship between the camera coordinate system and the vehicle coordinate system is dynamically adjusted according to the suspension state, avoiding projection deviations caused by changes in vehicle attitude.

[0046] Subsequently, the computing device can use the corrected extrinsic parameters to recalculate the projected coordinates of the feature points in the vehicle coordinate system. Specifically, the pixel coordinates (P1, P2, P3, …) of the feature points on the calibration reference are transformed into the vehicle coordinate system using the intrinsic parameters of the vehicle camera (such as focal lengths fx, fy and principal points cx, cy), distortion coefficients (k1, k2, k3, k4), and new extrinsic parameters (Rnew, tnew), generating updated projected coordinates (Hnew1, Hnew2, Hnew3, …). Based on these projected coordinates, the computing device can further calculate the corrected reprojection error of the feature points, i.e., the Euclidean distance between the projected coordinates and the true coordinates. This error quantifies the calibration accuracy of the corrected extrinsic parameters in a specific region.

[0047] Within the overlapping field of view of adjacent vehicle cameras, the computing device can re-execute the stitching line determination process based on the distribution of corrected reprojection errors. The specific principle is similar to that described above and will not be elaborated further. This method ensures that the stitching line position is dynamically optimized as the vehicle suspension changes, thereby maintaining the continuity and realism of the stitched image. Through this dynamic correction mechanism, this application not only solves the problem of suboptimal stitching positions caused by individual vehicle differences in factory calibration, but also avoids the interference of sudden stitching changes during operation on the user experience, improving the robustness and practicality of the panoramic imaging system. It ensures that each vehicle can obtain the best image stitching effect under different operating conditions.

[0048] This application first acquires calibration data, which may include the true coordinates of feature points of the calibration reference object in the vehicle coordinate system and the pixel coordinates on the camera image. Then, based on camera parameters, the projected coordinates of these pixel coordinates are calculated and reprojected into the vehicle coordinate system, and compared with the true coordinates to obtain the reprojection error of each feature point. This reprojection error directly reflects the accuracy difference of the camera model in different regions. Finally, within the overlapping area of ​​the field of view of adjacent vehicle cameras, a decision is made based on these calculated reprojection error values. Since the magnitude of the reprojection error directly reflects the accuracy of the image mapping, a target stitching line with the smallest reprojection error is determined. Using the target stitching line for image stitching can minimize stitching misalignment and ghosting caused by coordinate mapping deviations, thereby finding the optimal stitching position under the corresponding calibration data for each vehicle with different installation conditions, fundamentally overcoming the defect that the preset stitching line cannot adapt to individual differences.

[0049] Based on the above embodiments, the embodiments of this application can also use the target stitching line determined in each overlapping field of view to stitch together the images acquired by each vehicle camera.

[0050] For example, the computing device can establish a mapping relationship from the vehicle coordinate system to the pixel coordinates of each image based on the extrinsic and intrinsic parameters of the vehicle camera; For a target pixel within the overlapping field of view in the vehicle coordinate system, if the target pixel is located within a preset fusion zone centered on the stitching line, the pixel values ​​of the target pixel in the first image captured by the first camera and the second image captured by the second camera are obtained based on the mapping relationship. The two pixel values ​​are then weighted and fused, and the fusion result is used as the final pixel value of the target pixel. The weighting coefficient is determined by the relative position of the target pixel to the stitching line. If the target pixel is located outside the preset fusion zone, the corresponding final pixel value is determined from the first image or the second image based on the mapping relationship and the position of the target pixel relative to the target stitching line.

[0051] For example, computing devices can use coordinate mapping and adaptive fusion techniques to ensure seamless stitching in overlapping areas of view. The core of this approach lies in using the calibration parameters of the onboard cameras to establish a mapping relationship from the vehicle coordinate system to the pixel coordinates of each camera image.

[0052] Specifically, the computing device can construct a camera model based on the extrinsic and intrinsic parameters of the vehicle-mounted camera, as well as the distortion coefficients. This camera model first transforms the pixels in the vehicle coordinate system to the camera coordinate system using the extrinsic parameters, and then maps them to the pixel coordinates on the fisheye image using the intrinsic parameters and distortion coefficients, thereby completing a one-to-one mapping from the three-dimensional vehicle space to the RGB values ​​of the two-dimensional image pixels.

[0053] For any target pixel within the overlapping area of ​​the field of view in the vehicle coordinate system, the computing device can adopt different processing strategies based on its relative position to the target seam line. A blending zone can be preset around the seam line (for example, a range of 5° above and below the seam angle). If the target pixel (here denoted as point F) is located within this blending zone, a pixel fusion operation is performed: the computing device can use the above mapping relationship to obtain the pixel value RGB1 of point F in the first image captured by the first camera (such as the left view camera), and the pixel value RGB2 of point F in the second image captured by the second camera (such as the front view camera).

[0054] Subsequently, these two pixel values ​​can be weighted and fused, and the calculation formula can be:

[0055] Here, RGB represents the final pixel value, and the weight coefficient is not a fixed value, but is dynamically determined by the relative position of the target pixel F and the seam line.

[0056] In a specific embodiment of this application, the weight coefficient is positively correlated with the angle between the line connecting point F and the reference point (such as the lower right corner of the region) and the baseline edge (such as the bottom edge). This means that the closer point F is to the dominant region of a certain camera within the fusion band, the higher the weight of its corresponding image, thereby achieving a smooth transition.

[0057] If the target pixel is located outside the preset fusion zone, the computing device can directly select the pixel value from the corresponding camera image based on the position of the target pixel relative to the target stitching line. For example, the final pixel value of a point above the fusion area comes entirely from the forward-looking fisheye image, while the point below comes entirely from the left-looking fisheye image. Through this precise positioning based on mapping relationships and the gradual weighting strategy within the fusion zone, this application effectively eliminates the harsh boundaries at the stitching points of adjacent images, generating a visually coherent and natural panoramic bird's-eye view image.

[0058] Figure 3 This is a schematic diagram of the structure of an image stitching line determination device provided in an embodiment of this disclosure. This image stitching line determination device can be understood as a functional module of the aforementioned vehicle. For example... Figure 3 As shown, the image stitching line determining device 300 includes: The acquisition unit 310 is used to acquire the calibration data of the vehicle-mounted camera in the target vehicle. The calibration data includes the true coordinates of the feature points on the calibration reference in the vehicle coordinate system and the pixel coordinates on the camera image. The first calculation unit 320 is used to calculate the reprojection error of feature points based on calibration data. The reprojection error is used to characterize the deviation between the projected coordinates of the pixel coordinates mapped to the vehicle coordinate system through camera parameters and the real coordinates. The first determining unit 330 is used to determine the target stitching line for image stitching based on the reprojection error of feature points within the overlapping field of view of adjacent vehicle cameras.

[0059] In some possible implementations, the number of overlapping regions is multiple, and the first determining unit is specifically used for: Based on the reprojection error distribution of feature points within the overlapping area of ​​the target field of view, the target stitching line is determined with the optimization objective of minimizing the total reprojection error within the two sub-regions divided by the candidate stitching line; wherein, for any candidate stitching line, the corresponding total reprojection error is determined based on the reprojection error of feature points within each of the divided sub-regions.

[0060] In some possible implementations, the first determining unit is specifically used for: Within the overlapping area of ​​the target field of view, a reference point is determined. Starting from the reference point, multiple candidate seam lines are determined, and each candidate seam line corresponds to a direction angle. For each candidate seam line, the target visual overlap area is divided into a first sub-region and a second sub-region, with the candidate seam line as the boundary. The first projection error of feature points in the first sub-region is calculated using the camera parameters of the first camera, and the second projection error of feature points in the second sub-region is calculated using the camera parameters of the second camera. The corresponding total weight projection error is determined by using the first projection error and the second projection error of the same candidate seam line. From the multiple candidate seam lines, the candidate seam line with the smallest total weight projection error is determined as the target seam line.

[0061] In some possible implementations, the number of overlapping regions is multiple, and the first determining unit is specifically used for: Within the overlapping area of ​​the target field of view, the projected coordinates of each feature point are used as path nodes to determine the starting point, ending point, and connectivity rules between path nodes. Starting from the starting point, traverse the feasible paths that can reach the destination according to the connectivity rules; For each feasible path, the target visual overlap area is divided into a first sub-region and a second sub-region, with the feasible path as a candidate stitching line. The first projection error of feature points in the first sub-region is calculated using the camera parameters of the first camera, and the second projection error of feature points in the second sub-region is calculated using the camera parameters of the second camera. The corresponding total weight projection error is determined by using the first projection error and the second projection error of the same feasible path; From all feasible paths, the feasible path with the smallest total weight projection error is determined as the target seam line.

[0062] In some possible implementations, the device further includes: The stitching unit is used to stitch together the images acquired by each vehicle-mounted camera using the target stitching lines determined in each of the overlapping areas of the field of view.

[0063] In some possible implementations, the splicing unit is specifically used for: Based on the extrinsic and intrinsic parameters of the vehicle-mounted camera, a mapping relationship is established from the vehicle coordinate system to the pixel coordinates of each image. For a target pixel within the overlapping field of view in the vehicle coordinate system, if the target pixel is located within a preset fusion zone centered on the stitching line, the pixel value of the target pixel in the first image captured by the first camera and the second image captured by the second camera are obtained based on the mapping relationship. The two pixel values ​​are weighted and fused, and the fusion result is used as the final pixel value of the target pixel. The weight coefficient is determined by the relative position of the target pixel and the seam line. If the target pixel is located outside the preset fusion band, then based on the mapping relationship, the corresponding final pixel value is determined from the first image or the second image according to the position of the target pixel relative to the target stitching line.

[0064] In some possible implementations, the device further includes: The detection unit is used to detect the change in the vehicle suspension height of the target vehicle relative to the calibrated default height. The unit is used to update the extrinsic parameters of the vehicle camera based on the vehicle suspension height if the absolute value of the change is greater than or equal to a preset change threshold, so as to obtain the corrected extrinsic parameters. The first calculation unit is used to calculate the projected coordinates of the feature point in the vehicle coordinate system based on the corrected extrinsic parameters. The third calculation unit is used to calculate the corrected reprojection error of the feature points using the recalculated projection coordinates; The first determining unit is specifically used to determine the target stitching line for image stitching based on the corrected reprojection error within the overlapping area of ​​the field of view.

[0065] In some possible implementations, the calibration data also includes projected coordinates, and the acquisition unit is specifically used for: Acquire fisheye images of the vehicle-mounted camera, including calibration reference objects; Determine the pixel coordinates of feature points on the calibration reference from the fisheye image; Based on the true coordinates of the feature points in the vehicle coordinate system and the corresponding pixel coordinates, the extrinsic parameters of the vehicle camera are calculated. Using the intrinsic parameters, distortion coefficients, and extrinsic parameters of the vehicle-mounted camera, the pixel coordinates of the feature points are transformed into projected coordinates in the vehicle coordinate system.

[0066] This disclosure also provides a computing device, which includes: a memory storing a computer program; and a processor for executing the computer program, wherein when the computer program is executed by the processor, it can implement the methods of any of the above embodiments.

[0067] Example, Figure 4 This is a schematic diagram of the structure of a computing device according to an embodiment of this disclosure. See below for details. Figure 4The diagram illustrates a structural schematic suitable for implementing the computing device 1000 in the embodiments of this disclosure. The computing device 1000 in the embodiments of this disclosure may include, but is not limited to, devices such as in-vehicle terminals (e.g., in-vehicle navigation terminals), electronic control units (ECUs), etc. Figure 4 The computing device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0068] like Figure 4 As shown, the computing device 1000 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the computing device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0069] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows computing device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 A computing device 1000 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0070] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0071] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0072] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0073] The aforementioned computer-readable medium may be included in the aforementioned computing device; or it may exist independently and not assembled into the computing device.

[0074] The aforementioned computer-readable medium carries one or more programs that, when executed by the computing device, cause the computing device to: calculate the average power consumption corresponding to the nearest target distance traveled by the vehicle; receive a regional baseline power consumption and its upper and lower fluctuation range sent from the cloud within a predefined area; calculate an estimated dynamic energy consumption based on the average power consumption and the regional baseline power consumption, and calculate an estimated driving range based on the remaining available power and the estimated dynamic energy consumption, wherein the estimated dynamic energy consumption conforms to the upper and lower fluctuation range; and update the displayed driving range value based on the estimated driving range.

[0075] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0076] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0077] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0078] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0079] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0080] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0082] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining stitching lines in an image, characterized in that, include: Acquire calibration data of the vehicle-mounted camera in the target vehicle. The calibration data includes the true coordinates of feature points on the calibration reference in the vehicle coordinate system and the pixel coordinates on the camera image. Based on the calibration data, the reprojection error of the feature point is calculated. The reprojection error is used to characterize the deviation between the projected coordinates of the pixel coordinates mapped to the vehicle coordinate system through camera parameters and the real coordinates. Within the overlapping field of view of adjacent vehicle cameras, the target stitching line for image stitching is determined based on the reprojection error of the feature points.

2. The method according to claim 1, characterized in that, The number of overlapping visual fields is multiple. Within the target overlapping visual fields of these multiple overlapping visual fields, the target stitching line for image stitching is determined based on the reprojection error of the feature points, including: Based on the reprojection error distribution of the feature points within the overlapping area of ​​the target field of view, the target stitching line is determined with the optimization objective of minimizing the total reprojection error within the two sub-regions divided by the candidate stitching line; wherein, for any candidate stitching line, the corresponding total reprojection error is determined based on the reprojection error of the feature points within each sub-region.

3. The method according to claim 2, characterized in that, Based on the reprojection error distribution of the feature points within the overlapping area of ​​the target field of view, and with minimizing the total reprojection error as the optimization objective, the target stitching line is determined, including: Within the overlapping area of ​​the target field of view, a reference point is determined. Starting from the reference point, multiple candidate seam lines are determined, and each candidate seam line corresponds to a direction angle. For each candidate seam line, the target visual overlap area is divided into a first sub-region and a second sub-region, with the candidate seam line as the boundary. The first projection error of feature points in the first sub-region is calculated using the camera parameters of the first camera, and the second projection error of feature points in the second sub-region is calculated using the camera parameters of the second camera. The corresponding total weight projection error is determined by using the first projection error and the second projection error of the same candidate seam line. From the multiple candidate seam lines, the candidate seam line with the smallest total weight projection error is determined as the target seam line.

4. The method according to claim 1, characterized in that, The number of overlapping visual fields is multiple. Within the target overlapping visual fields of these multiple overlapping visual fields, the target stitching line for image stitching is determined based on the reprojection error of the feature points, including: Within the overlapping area of ​​the target field of view, the projected coordinates of each feature point are used as path nodes to determine the starting point, ending point, and connectivity rules between path nodes. Starting from the starting point, traverse the feasible paths that can reach the destination according to the connectivity rules; For each feasible path, the target visual overlap area is divided into a first sub-region and a second sub-region, with the feasible path as a candidate stitching line. The first projection error of feature points in the first sub-region is calculated using the camera parameters of the first camera, and the second projection error of feature points in the second sub-region is calculated using the camera parameters of the second camera. The corresponding total weight projection error is determined by using the first projection error and the second projection error of the same feasible path; From all feasible paths, the feasible path with the smallest total weight projection error is determined as the target seam line.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Images acquired by each vehicle-mounted camera are stitched together using the target stitching lines determined within each overlapping field of view region.

6. The method according to claim 5, characterized in that, The step of stitching together images captured by each vehicle-mounted camera using target stitching lines determined within each overlapping field of view includes: Based on the extrinsic and intrinsic parameters of the vehicle-mounted camera, a mapping relationship is established from the vehicle coordinate system to the pixel coordinates of each image. For a target pixel within the overlapping field of view in the vehicle coordinate system, if the target pixel is located within a preset fusion zone centered on the stitching line, the pixel value of the target pixel in the first image captured by the first camera and the second image captured by the second camera are obtained based on the mapping relationship. The two pixel values ​​are weighted and fused, and the fusion result is used as the final pixel value of the target pixel. The weight coefficient is determined by the relative position of the target pixel and the seam line. If the target pixel is located outside the preset fusion band, then based on the mapping relationship, the corresponding final pixel value is determined from the first image or the second image according to the position of the target pixel relative to the target stitching line.

7. The method according to claim 1, characterized in that, The method further includes: Detect the change in the vehicle suspension height of the target vehicle relative to the calibrated default height; If the absolute value of the change is greater than or equal to a preset change threshold, the extrinsic parameters of the vehicle camera are updated according to the vehicle suspension height to obtain the corrected extrinsic parameters. Based on the corrected extrinsic parameters, calculate the projected coordinates of the feature point in the vehicle coordinate system; The corrected reprojection error of the feature point is calculated using the recalculated projection coordinates. Within the overlapping field of view of adjacent vehicle cameras, based on the reprojection error of the feature points, the target stitching line for image stitching is determined, including: Within the overlapping area of ​​the field of view, the target stitching line for image stitching is determined based on the corrected reprojection error.

8. The method according to claim 1, characterized in that, The calibration data also includes the projected coordinates, and the acquisition of calibration data for the vehicle-mounted camera in the target vehicle includes: Acquire fisheye images of the vehicle-mounted camera, including calibration reference objects; Determine the pixel coordinates of feature points on the calibration reference from the fisheye image; Based on the true coordinates of the feature points in the vehicle coordinate system and the corresponding pixel coordinates, the extrinsic parameters of the vehicle camera are calculated. Using the intrinsic parameters, distortion coefficients, and extrinsic parameters of the vehicle-mounted camera, the pixel coordinates of the feature points are transformed into projected coordinates in the vehicle coordinate system.

9. A computing device, characterized in that, The computing device includes: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-8.

10. A vehicle, characterized in that, Includes the computing device as described in claim 9.

Images