Panoramic image stitching method and device, equipment and medium

Abstract

The invention discloses a panoramic image splicing method, device and equipment and a medium, and relates to an intelligent cockpit, and the method comprises the steps: determining the image splicing precision according to the hardware computing power of a target vehicle and the application scene of a panoramic image; according to the image splicing precision, obtaining target mapping data from total mapping data pre-deployed in the target vehicle, the total mapping data including a total mapping relationship of each splicing pixel point in a panoramic image with respect to M paths of surround view cameras, the total mapping relations of the same spliced pixel point with respect to the M paths of all-round view cameras are ranked according to the mapping weights; and acquiring M paths of original images acquired by M paths of surround view cameras installed on the target vehicle at the current moment, and splicing the M paths of original images according to the target mapping data to obtain a current frame of panoramic image. According to the invention, the technical problem that the precision of the spliced panoramic image cannot be guaranteed in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of smart cockpits, and in particular to a panoramic image stitching method, device, equipment and medium. Background Art

[0002] With the growing demand for automotive intelligence and the improvement of the camera access capabilities of vehicle-mounted domain controllers, panoramic image perspectives (such as real-time 360-degree surround view display) and their derivative applications (parking space recognition, APA autonomous parking) have become increasingly popular and even become standard features of vehicles. In these applications, it is inevitable to stitch the original images exposed by multiple surround view cameras with a common field of view into the same picture. Panoramic stitching uses the AVM (Around View Monitor, panoramic surround view system) bird's-eye view. The top-down view can better show the driver the surrounding environment and provide a clear basis for judgment in terms of direction. It can assist manual driving to smoothly pass through various scenes such as narrow roads and parking spaces without limiters, etc. It can also provide transparent chassis display, APA automatic parking, lane departure warning / attitude maintenance, etc. to the autonomous driving system.

[0003] Due to the wide range of uses and the uncertainty of the platform, the original images exposed by multiple surround view cameras at the same time are stitched into the same bird's-eye view map. Common existing technologies include: (1) real-time stitching of multiple exposure images through matching algorithms; (2) generating a stitching lookup table through camera model relationships. The real-time stitching of multiple exposure images completed through matching algorithms will be greatly affected by the external environment and it is difficult to provide a true bird's-eye view; the stitching lookup table generated through the camera model relationship is prone to loss of accuracy when stitching AVM images and lacks the ability to adapt to different scenes. In other words, the accuracy of panoramic images stitched using existing technologies cannot be guaranteed. Summary of the invention

[0004] The embodiments of the present specification provide a panoramic image stitching method, device, equipment and medium, which are used to solve the technical problem that the accuracy of the panoramic images stitched in the prior art cannot be guaranteed.

[0005] In a first aspect of the present invention, a method for stitching a panoramic image is provided, comprising: determining an image stitching accuracy according to a hardware computing power of a target vehicle and an application scenario of the panoramic image; acquiring target mapping data from full mapping data pre-deployed on the target vehicle according to the image stitching accuracy, the full mapping data comprising a full mapping relationship of each stitching pixel point in the panoramic image with respect to M-way surround view cameras, and the full mapping relationship of the same stitching pixel point with respect to the M-way surround view cameras is sorted according to a mapping weight, where M is an integer greater than 1; acquiring M-way original images collected by the M-way surround view cameras installed on the target vehicle at a current moment, and stitching the M-way original images according to the target mapping data to obtain a current frame panoramic image.

[0006] In combination with the first aspect, in some embodiments, the step of pre-generating the full mapping data is also included: determining the parameter calibration results of M-way surround-view cameras installed on the target vehicle and the area occupied by the target vehicle in the panoramic image; generating an image stitching function entrance according to the parameter calibration results and the occupied area; generating M grayscale images matching the imaging size of the M-way surround-view cameras according to the parameter calibration results; and processing the M grayscale images according to the image stitching function entrance to obtain the full mapping data.

[0007] In combination with the first aspect, in some embodiments, determining the parameter calibration results of M-way surround-view cameras installed on the target vehicle includes: calibrating the camera intrinsic parameters and camera extrinsic parameters of each of the M-way surround-view cameras to obtain the internal and external parameter calibration results of the M-way surround-view cameras; and determining the mapping optimization parameters for the overlapping imaging area of ​​each two adjacent surround-view cameras in the M-way surround-view cameras based on the internal and external parameter calibration results.

[0008] In combination with the first aspect, in some embodiments, determining the mapping optimization parameters for the overlapping imaging area of ​​each two adjacent surround-view cameras in the M surround-view cameras based on the internal and external parameter calibration results includes: determining the overlapping imaging area of ​​each two adjacent surround-view cameras in the M surround-view cameras based on the internal and external parameter calibration results to obtain M overlapping imaging areas; and determining the color weight of the pixel with respect to the two adjacent surround-view cameras for each pixel point in each overlapping imaging area in the M overlapping imaging areas.

[0009] In combination with the first aspect, in some embodiments, determining the color weight of the pixel point with respect to two adjacent surround-view cameras includes: determining the angular offset between a line connecting the pixel point and the center point of the area occupied by the target vehicle in the panoramic image and the central axis of the two adjacent surround-view cameras to obtain two angular offsets of the pixel point with respect to the two adjacent surround-view cameras; and normalizing the two angular offsets of the pixel point with respect to the two adjacent surround-view cameras to obtain two color weights of the pixel point with respect to the two adjacent surround-view cameras.

[0010] In combination with the first aspect, in some embodiments, generating M grayscale images matching the imaging size of the M surround-view cameras according to the parameter calibration results includes: for the M surround-view cameras installed on the target vehicle, generating a grayscale image equal to the imaging size of the surround-view camera and inheriting the internal and external calibration parameters of the surround-view camera according to the internal and external parameter calibration results of the surround-view camera, wherein the pixel value of each pixel of the grayscale image is zero.

[0011] In combination with the first aspect, in some embodiments, the M grayscale images are processed according to the image stitching function entrance to obtain the full mapping data, including: traversing the pixels of the M grayscale images one by one; for each traversed pixel, modifying the grayscale value of the pixel currently traversed to obtain a new grayscale image, stitching the new grayscale image with other M-1 grayscale images to obtain a current stitched image, and determining the mapping data of each target pixel in the current stitched image, wherein the pixel value of the target pixel is not zero; after the traversal of the M grayscale images is completed, based on the mapping data obtained by modifying a pixel in the M grayscale images each time, the full mapping relationship of each stitched pixel is combined; for each stitched pixel, the full mapping relationship belonging to the stitched pixel is sorted from large to small according to the mapping weight to obtain the full mapping data.

[0012] In combination with the first aspect, in some embodiments, the full mapping relationship of each stitched pixel point includes: in the original image exposed by M-way surround view cameras, multi-dimensional information of each original pixel point will be mapped to the stitched pixel point, and the multi-dimensional information of each original pixel point includes the coordinate information of the original pixel point, the source camera ID of the surround view camera providing the original pixel point, and the mapping weight of the original pixel point to the stitched pixel point.

[0013] In a second aspect of the present invention, a panoramic image stitching device is provided, comprising: an accuracy determination unit, for determining the image stitching accuracy according to the hardware computing power of the target vehicle and the application scenario of the panoramic image; a data determination unit, for obtaining target mapping data from full mapping data pre-deployed on the target vehicle according to the image stitching accuracy, the full mapping data including the full mapping relationship of each stitching pixel point in the panoramic image with respect to M-way surround view cameras, and the full mapping relationship of the same stitching pixel point with respect to the M-way surround view cameras is sorted according to the mapping weight, M being an integer greater than 1; an image stitching unit, for obtaining M-way original images collected by the M-way surround view cameras installed on the target vehicle at the current moment, and stitching the M-way original images according to the target mapping data to obtain a current frame panoramic image.

[0014] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for stitching panoramic images described in any one of the embodiments of the first aspect is implemented.

[0015] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for stitching panoramic images described in any embodiment of the first aspect is implemented.

[0016] One or more technical solutions provided by the embodiments of this specification have at least the following technical effects or advantages:

[0017] Since the present embodiment determines the image stitching accuracy according to the hardware computing power of the target vehicle and the application scenario of the panoramic image; according to the image stitching accuracy, the target mapping data is obtained from the full mapping data pre-deployed on the target vehicle, and the full mapping data includes the full mapping relationship of each stitching pixel point in the panoramic image with respect to the M-way surround view camera, and the full mapping relationship of the same stitching pixel point with respect to the M-way surround view camera is sorted according to the mapping weight; after obtaining the M-way original images collected by the M-way surround view camera installed on the target vehicle at the current moment, the M-way original images can be stitched according to the target mapping data to obtain the current frame panoramic image. This solution is simple and efficient, does not require the use of other auxiliary equipment, and does not require engineers to manually perform complex camera models and three-dimensional space conversion calculations. It only uses the pre-generated full mapping data, and can choose to use part or all of the mapping data in the full mapping data according to the required image stitching accuracy when stitching the panoramic image later, and can stitch out a panoramic image with an accuracy that matches the actual application scenario and hardware computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0019] Figure 1 A flow chart of a method for stitching panoramic images in an embodiment of this specification is shown;

[0020] Figure 2 A schematic diagram of image stitching of four surround view cameras installed on a vehicle in an embodiment of this specification is shown;

[0021] FIG. 3( a ) shows a schematic diagram of the mapping relationship of the first LUT file before sorting in the embodiment of this specification;

[0022] FIG. 3( b ) shows a schematic diagram of the sorted mapping relationship of the first LUT file in the embodiment of this specification;

[0023] Figure 4 A schematic diagram of the structure of a panoramic image stitching device in an embodiment of this specification is shown;

[0024] Figure 5 A schematic diagram of the structure of an electronic device in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0025] In order to better understand the above technical scheme, the technical scheme of the embodiments of this specification is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical scheme of the embodiments of this specification, rather than limitations on the technical scheme of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0026] The present invention provides a method for stitching panoramic images. Figure 1 As shown, the panoramic image stitching method includes the following steps S101 to S103:

[0027] S101. Determine image stitching accuracy according to the hardware computing power of the target vehicle and the application scenario of the panoramic image.

[0028] It is understandable that the application scenarios of panoramic images may include: providing panoramic images to the driver for viewing, providing them to the upper-level algorithm application to assist manual driving, or providing them to the autonomous driving system. The panoramic images provided to the driver do not require too high image stitching accuracy, while the panoramic images provided to the upper-level algorithm application and to the autonomous driving system require higher image stitching accuracy than the panoramic images provided to the driver. The higher the image stitching accuracy, the higher the requirements for the vehicle's hardware computing power. Therefore, it is necessary to determine the image stitching accuracy in combination with the hardware computing power of the target vehicle and the application scenario of the panoramic image. The hardware computing power of the target vehicle includes the computing power of the SOC (System on Chip) chip of the target vehicle's on-board hardware platform.

[0029] S102. According to the image stitching accuracy, target mapping data is obtained from the full mapping data pre-deployed on the target vehicle, where the full mapping data includes a full mapping relationship of each stitching pixel point in the panoramic image with respect to M surround view cameras, and the full mapping relationships of the same stitching pixel point with respect to the M surround view cameras are sorted according to mapping weights, where M is an integer greater than 1.

[0030] It is understandable that when the panoramic image stitching function of the target vehicle is turned on, the M-way surround view cameras installed on the target vehicle will generate M original images at the same time, and multiple pixels from the M original images will be mapped to the same stitched pixel on the panoramic image. The full mapping data includes the full mapping relationship of each stitched pixel in the panoramic image with respect to the M-way surround view cameras, and the full mapping relationship of each stitched pixel with respect to the M-way surround view cameras includes: the multi-dimensional information of each original pixel in the original image exposed by the M-way surround view cameras that will be mapped to the stitched pixel, and the multi-dimensional information of each original pixel includes the coordinate information of the original pixel, the source camera ID of the surround view camera that provides the original pixel, and the mapping weight of the original pixel mapped to the stitched pixel.

[0031] In the full mapping relationship of the same stitching pixel point with respect to M-way surround view cameras, each group of mapping relationships is sorted from large to small according to the mapping weight, and the image stitching accuracy determines the number of mapping relationship groups required to calculate the color value of each stitching pixel point when stitching the original images exposed by the M-way surround view cameras into a panoramic image. The higher the image stitching accuracy, the more groups of mapping relationships for each stitching pixel point in the obtained target mapping data, making the stitched panoramic image more realistic.

[0032] Read the first LUT file containing the full mapping data into the memory space, and pre-process the first LUT file according to the image stitching accuracy to obtain the second LUT file containing the target mapping data, and write the obtained second LUT file into the memory according to the set logic for subsequent use in stitching panoramic images.

[0033] Among them, for the full mapping relationship of each spliced ​​pixel, read the N groups of mapping relationships in the full mapping relationship of the spliced ​​pixel according to the mapping weight from large to small; for each N group of mapping relationships of the spliced ​​pixel, normalization processing is performed respectively to obtain the target mapping data, wherein the higher the image splicing accuracy, the larger the N value. If the panoramic image is to be spliced ​​for the driver to watch, the N value can be 1 or 2, that is, in the full mapping relationship of each spliced ​​pixel, only the group of mapping relationships with the largest mapping weight or the two groups of mapping relationships with the largest and second largest mapping weights are taken to obtain the target mapping data.

[0034] Taking the lookup table data in the first LUT file shown in FIG3(b) as an example, there are only three sets of mapping relationships for the full mapping relationship of the stitching pixel point at row 0 and column 0 in the panoramic image, and their data are [0,458,318,0.66, 1,458,319,0.33, 0,459,318,0.01]. If the number of mapping groups determined by the image stitching accuracy is 3 or more, then these three sets of mapping relationships will be read into the memory space normally and used to calculate the pixel value of the stitching pixel point at row 0 and column 0 in the panoramic image; if the number of mapping groups determined by the image stitching accuracy is 3 or more, then these three sets of mapping relationships will be read into the memory space normally and used to calculate the pixel value of the stitching pixel point at row 0 and column 0 in the panoramic image; If the number of mapping groups determined by the connection accuracy is 2, then only the two mapping relationships [0,458,318,0.66, 1,458,319,0.33] with the mapping weights arranged in the front will be read, and the mapping weights 0.66 and 0.33 will be renormalized to 0.6666 and 0.3333 (4 decimal places can be set to be retained). Thus, the preprocessed mapping relationship becomes [0,458,318,0.6666,1,458,319,0.3333], which is used to calculate the pixel value of the spliced ​​pixel point at row 0 and column 0 in the panoramic image.

[0035] In some implementations, the present illustrative embodiment further includes the following steps S201 to S203 of pre-generating full mapping data.

[0036] S201, determining parameter calibration results of M-way surround view cameras installed on a target vehicle and an area occupied by the target vehicle in a panoramic image.

[0037] The parameter calibration result in step S201 may include the internal and external parameter calibration results of the M-way surround view cameras and the mapping optimization parameters for the overlapping imaging area of ​​each two adjacent surround view cameras in the M-way surround view cameras. In some embodiments, the camera internal parameters and camera external parameters of each surround view camera in the M-way surround view cameras are calibrated to obtain the internal and external parameter calibration results of the M-way surround view cameras; according to the internal and external parameter calibration results, the mapping optimization parameters suitable for the overlapping imaging area of ​​each two adjacent surround view cameras in the M-way surround view cameras are determined. In addition to the ordinary camera coordinate conversion, the optimization operation of the overlapping area is also considered in the full mapping relationship, so that a more natural and realistic panoramic image can be spliced.

[0038] In the embodiments of this specification, the surround view camera may be a fisheye camera. The fisheye camera has a large wide-angle characteristic and can achieve 360-degree panoramic field of view coverage at a close distance with as few cameras as possible. The fisheye camera is a type of camera commonly used in AVM images.

[0039] It should be noted that the camera intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc., and the camera extrinsic parameters are parameters in the world coordinate system, such as the camera's position, rotation direction, etc. Figure 2 As shown, four surround view cameras are usually installed corresponding to the front, rear, left and right sides of the vehicle. In step S201, the camera intrinsic parameters and camera extrinsic parameters of the four surround view cameras installed on the target vehicle are calibrated respectively to obtain the internal and external parameter calibration results of each surround view camera. The calibration method and the format of generating the internal and external parameter calibration results are not limited, and only the parameter information of mapping the original images exposed by the four surround view cameras to the same panoramic image is required. Of course, in the actual implementation process, it is not limited to installing only four surround view cameras.

[0040] In some embodiments, the mapping optimization parameters applicable to the overlapping imaging area of ​​each adjacent two-way surround-view camera include: the color weight of each pixel point in each overlapping imaging area with respect to the corresponding adjacent two-way surround-view camera, the corresponding adjacent two-way surround-view camera is the two surround-view cameras that form the overlapping imaging area, therefore, for M overlapping imaging areas formed by M-way surround-view cameras, it is necessary to calculate the color weight of each pixel point in each overlapping imaging area with respect to the corresponding adjacent two-way surround-view camera.

[0041] like Figure 2As shown in the figure, taking the installation of four surround view cameras corresponding to the front, rear, left and right of the vehicle as an example, the original images exposed by the four surround view cameras are spliced ​​into the panoramic image in the same coordinate system, and there will be four overlapping imaging areas A1, A2, A3 and A4. For each overlapping imaging area, the color weight of each pixel in the overlapping imaging area with respect to the corresponding two adjacent surround view cameras is calculated. In actual operation, the entire image area formed by splicing the original images exposed by each surround view camera into the panoramic image is cut into multiple rectangular image areas. Therefore, the overlapping imaging area, non-overlapping imaging area and vehicle body occupied area of ​​any two adjacent surround view cameras are all rectangular.

[0042] In some embodiments, mapping optimization parameters for the overlapping imaging area of ​​each two adjacent surround view cameras in M-way surround view cameras are determined based on the internal and external parameter calibration results, including: determining the overlapping imaging area of ​​each two adjacent surround view cameras in the M-way surround view cameras based on the internal and external parameter calibration results to obtain M overlapping imaging areas; for each pixel point in each overlapping imaging area of ​​the M overlapping imaging areas, determining the color weight of the pixel point with respect to the two adjacent surround view cameras.

[0043] It is understandable that there are many ways to calculate the color weight of each pixel point in each overlapping imaging area with respect to the corresponding two adjacent surround-view cameras, which can be determined by the surround-view camera. In some embodiments, for each pixel point in each overlapping imaging area of ​​the M overlapping imaging areas, the angle offset between the line connecting the pixel point and the center point of the area occupied by the target vehicle in the panoramic image and the central axis of the two adjacent surround-view cameras is determined to obtain the two angle offsets of the pixel point with respect to the corresponding two adjacent surround-view cameras, and the two angle offsets of the pixel point with respect to the corresponding two adjacent surround-view cameras are normalized to obtain the color weight of the pixel point with respect to the corresponding two adjacent surround-view cameras. It should be noted that the two adjacent surround-view cameras corresponding to each pixel point are the two surround-view cameras that form the pixel point.

[0044] For example, the closer the surround view camera is to the edge, the more serious the distortion is. Even after de-distortion, there is still a certain degree of distortion. Therefore, for each pixel point in each overlapping imaging area in the panoramic image, the angle offset between the line connecting the pixel point to the center position of the target vehicle and the central axis of the two adjacent surround view cameras is calculated. Taking a pixel point X in a certain overlapping imaging area as an example, the following can be obtained: Figure 2The angle offset ∠A and the angle offset ∠B shown in , the two angle offsets ∠A and ∠B are normalized, that is, the two color weights of the pixel point with respect to the two adjacent surround view cameras are obtained. The larger the angle offset, the smaller the color weight of the original pixel point of the corresponding surround view camera in the overlapping imaging area, which means that the color of the original pixel point of the original image exposed by the surround view camera is projected to the overlapping imaging area of ​​the panoramic image. For example: if the angle offset ∠A is 60° and the angle offset ∠B is 30°, then the two color weights of the pixel point with respect to the two adjacent surround view cameras are 60 / 90=2 / 3, 30 / 90=1 / 3.

[0045] S202: Generate a stitching function entry according to the parameter calibration result and the area occupied by the target vehicle in the panoramic image.

[0046] In some embodiments, an image stitching function entrance is generated based on the internal and external parameter calibration results of the M-way surround view cameras, the mapping optimization parameters applicable to the overlapping imaging area of ​​each two adjacent surround view cameras in the M-way surround view cameras, and the area occupied by the target vehicle in the panoramic image, so that the image stitching function entrance can identify each pixel point occupied by the target vehicle in the panoramic image and each pixel point belonging to the overlapping imaging area in the panoramic image.

[0047] S203 . Generate M grayscale images that match the imaging size of the M-way surround view cameras according to the internal and external parameter calibration results.

[0048] For the M surround-view cameras installed on the target vehicle, a grayscale image having an imaging size equal to that of the surround-view camera and inheriting the internal and external calibration parameters of the surround-view camera is generated according to the internal and external calibration results of the surround-view cameras. In the generated M grayscale images, the pixel value of each pixel in each grayscale image is zero, that is, the M grayscale images are all pure black grayscale images.

[0049] The grayscale image with the same imaging size as the surround view camera is generated with the same height and width as the original image exposed by the surround view camera. A grayscale image represents a surround view camera and inherits the internal and external calibration results of the surround view camera. The grayscale image number range is [0, M-1]).

[0050] S204, processing the M grayscale images according to the image stitching function entry to obtain full mapping data.

[0051] In some implementations, step S204 may include the following steps S2041 to S2043:

[0052] S2041. Traverse the pixels of the M grayscale images one by one through the image stitching function entrance.

[0053] In step S2041, the pixels of the M grayscale images are traversed one by one according to the source camera ID of the M surround view cameras, the coordinate row ID of the original pixel point (the range of the coordinate row ID is [0 to the imaging height of the surround view camera]), and the coordinate column ID of the original pixel point (the range of the coordinate column ID is [0 to the imaging width of the surround view camera)).

[0054] S2042. For each pixel point traversed each time, modify the grayscale value of the pixel point traversed currently to obtain a new grayscale image, and splice the new grayscale image with other M-1 grayscale images to obtain a current spliced ​​image, and determine the mapping data of each target pixel point in the current spliced ​​image, and the pixel value of the target pixel point is not zero.

[0055] It should be noted that each time only one pixel is traversed from the M grayscale images and the pixel value of the pixel is modified to 255, and the pixel value of the pixel traversed last time will be restored to 0. Therefore, among the M grayscale images used to obtain the current stitched image, only the grayscale value of the pixel traversed currently is 255, and the pixel values ​​of other pixels in the M grayscale images are all 0.

[0056] In some embodiments, after each stitching to obtain the current stitched image, each stitched pixel in the current stitched image is traversed row by row and column by column to determine each stitched pixel whose pixel value is not zero in the current stitched image, that is, each target pixel in the current stitched image is obtained. If the gray value (expressed as G_value) of the stitched pixel traversed from the current stitched image is not 0, it means that there is a mapping relationship between the stitched pixel and the original pixel whose gray value is modified to 255 in the M grayscale images, that is, the stitched pixel is the target pixel, and the mapping weight of the original pixel whose gray value is modified to 255 mapped to the stitched pixel is G_value / 255.0.

[0057] It can be understood that the mapping data of each target pixel in the current stitched image includes: the pixel coordinates of the target pixel, the coordinates of the original pixel in the grayscale image that has a mapping relationship with the target pixel, the source camera ID of the corresponding surround camera, and the mapping weight. After the current stitched image is traversed row by row and column by column to obtain the mapping data of each target pixel in the current stitched image, the pixel value of the pixel whose pixel value is currently modified to 255 in the M grayscale images needs to be restored to 0.

[0058] S2043. After the traversal of the M grayscale images is completed, based on the mapping data obtained by modifying a pixel point in the M grayscale images each time, a full mapping relationship of each spliced ​​pixel point is obtained by combining them.

[0059] Among them, each stitched pixel point can establish a mapping relationship with dozens or even more original pixel points of the original image exposed by the surround view camera. The higher the value of the mapping weight contained in each set of mapping relationships, the greater the proportion of the original pixel points of the original image mapped to the stitched pixel points in the set of mapping relationships.

[0060] Taking a panoramic image containing 512x512 stitching pixels as an example, the mapping relationships of each group of stitching pixels obtained by combining are shown in FIG3(a), and there are 512x512 stitching pixels in total.

[0061] S2044. For each spliced ​​pixel point, sort the full mapping relationship of the spliced ​​pixel point according to the mapping weight from large to small to obtain full mapping data.

[0062] In some embodiments, the step of pre-generating full mapping data may also include: storing the full mapping data as a first LUT (Look-up Table) file in the form of a medium such as a table file. In the embodiment of this specification, the first LUT file is used for pixel-level conversion of the original image exposed by the surround camera to the panoramic image.

[0063] As shown in Figure 3(b), the LUT file of the full mapping data required for a panoramic image of size 512x512 is displayed. In addition to the subject name "avm_lut", the first-level nodes 0 to 511 represent the row coordinates of the panoramic image. Each first-level node has a second-level node composed of 0 to 511, representing the column coordinates of the panoramic image. The specific content format under the second-level node is: [(mapping relationship 1) source camera ID, row coordinates of the original image, column coordinates of the original image, mapping weight, (mapping relationship 2) source camera ID, row coordinates of the original image, column coordinates of the original image, mapping weight, ...].

[0064] As shown in Figure 3(b), the data [0,458,318,0.66,1,458,319,0.33,0,459,318,0.01] in [avm_lut][0][0] means that the stitched pixel point at row 0 and column 0 in the panoramic image contains three groups of mapping relationships with the surround view camera, among which the mapping weight of the pixel point at row 458 and column 318 in the original image exposed by the surround view camera 0 is 0.66, the mapping weight of the pixel point at row 458 and column 319 in the original image exposed by the surround view camera 1 is 0.33, and the mapping weight of the pixel point at row 459 and column 318 in the original image exposed by the surround view camera 0 is 0.01. It is worth mentioning that for each stitched pixel point in the panoramic image, the sum of the mapping weights of all the groups of mapping relationships belonging to the stitched pixel is 1.

[0065] S103, obtaining M-way original images collected by M-way surround view cameras installed on the target vehicle at the current moment, and splicing the M-way original images according to the target mapping data to obtain a current frame panoramic image.

[0066] When the panoramic stitching function of the target vehicle is turned on, the on-board domain controller will synchronously trigger the M-way surround view cameras to collect the M-way original images exposed at the same time; the second LUT table containing the target mapping data obtained in step S102 is used to stitch the M-way original images exposed at the same time to obtain the current frame panoramic image.

[0067] In some embodiments, the M-way original images captured by the M-way surround view camera may be color images of a certain color space, for example, may be color images of a BGR space, then the M-way original images include M-way image data of each color channel in a plurality of color channels, and for each color channel, the target mapping data is used to splice the M-way image data of the color channel to obtain the pixel value of each spliced ​​pixel point on the color channel, thus obtaining a spliced ​​image of the color channel, wherein, when obtaining the pixel value of each spliced ​​pixel point on the color channel, the pixel value of each pixel point of the original image mapped by the spliced ​​pixel point is located according to the starting address of the M-way image data in the cache, and then a weighted sum is performed according to the mapping weight value of the spliced ​​pixel point in the second LUT lookup table to obtain the pixel value of the spliced ​​pixel point on the color channel, and finally the spliced ​​images of each color channel are merged to generate a panoramic image of the corresponding color space.

[0068] In some embodiments, it is not necessary to directly generate an image file of a panoramic image. Instead, the image data of the panoramic image is stored in the form of a buffer and directly provided to an upper-level application. For example, when a deep learning model is used to detect parking spaces, the data can be directly written into the cache area to be inferred in a specified order without generating an image file. This can further save the generation of image files and the conversion operation of the image file into image data that can be used by the upper-level application. As a result, the related three-party dependencies can be removed.

[0069] Thanks to the leap in chip capabilities and the gradual adaptation of domain controllers and cameras of multiple brands, it is technically mature to arrange fisheye cameras around the vehicle body and provide surround view stitching in real time, but it often requires expensive software and hardware cost support.

[0070] When the fisheye camera is installed on the vehicle, under the premise of calibration and confirmation of the field of view of the panoramic image, the coordinate relationship between each original pixel in the original image exposed by each fisheye camera and mapped to the panoramic image is constant (without considering the optimization processing such as white balance after the panoramic image is generated).

[0071] The embodiment of this specification makes use of this feature. After the camera calibration is completed, through steps S2041 to S2044, the mapping relationship between each original pixel on each original image and the spliced ​​pixel on the panoramic image is directly calculated and generated in a sliding window manner, and the mapping weights are arranged and saved in the first LUT file. In addition to the ordinary camera coordinate conversion, the optimization operation of the overlapping imaging area is also considered. Therefore, a more natural and realistic panoramic image can be spliced. After the first LUT file is made, the vehicle will read the full amount of mapping data in the first LUT file in advance during actual operation. The full amount of mapping data read in is selectively preprocessed according to the actual computing power of the vehicle's SOC and the application scenario. After the panoramic image stitching function is started, the pixel value of the specified pixel coordinate can be quickly searched from the original image of the four-way surround view camera according to the preprocessed target mapping data to fill in the panoramic image.

[0072] One or more implementations provided in the embodiments of this specification utilize the characteristic that "the coordinate relationship of each original pixel point in the original image formed by each fisheye camera mapped to the panoramic image is constant", and realize a first LUT table with full-precision mapping relationship and its production process, which is almost unaffected by external factors such as camera model replacement. The production of the lookup table is simple but rich and effective, and even includes factors such as stitching optimization of adjacent surround-view cameras. It also innovatively proposes the concept of using lookup table data with variable precision, which greatly enhances versatility. This solution has a low threshold for operation and a high level of automation. While ensuring high performance and low cost, it also provides more options for image stitching accuracy. It puts the time-consuming items into two time periods: vehicle data preparation (the process of pre-producing the full amount of mapping data before installation) and vehicle startup (before the panoramic image stitching function is started, the target mapping data is determined based on the hardware computing power and application scenarios). It has extremely low dependence on the computing resources of the on-board SOC chip, and can take into account both high performance and low-cost hardware chip environments. It also provides a balanced choice of resources and image stitching accuracy, so that the appropriate stitching accuracy can be selected after a reasonable assessment of the vehicle's SOC chip capabilities and application scenarios. This solution does not need to rely on other auxiliary equipment, nor does it require professional cameras and knowledge reserves in the field of three-dimensional space. It greatly reduces the deployment threshold of panoramic image stitching and its derivative applications. Therefore, it can be applied to vehicles using medium and low-cost computing platforms.

[0073] Compared with the solution of completing the real-time stitching of multiple images through matching algorithms, the embodiments of this specification are still completed under the premise / framework of pre-calibration of internal and external parameters, which can provide the required bird's-eye view and complete accurate surround stitching in various environments. There is no need to develop and test related alignment algorithms, and the stitching effect is not affected by external factors. Compared with the solution of generating a stitching lookup table through the camera model relationship, the embodiments of this specification do not require R&D personnel to have professional knowledge of the camera imaging principle and the pixel-three-dimensional space point internal and external parameter transformation. The transformation of the camera model does not affect the current process. It has strong adaptability, and in the generated first LUT table containing the full amount of mapping data, each stitching pixel contains all the mapping relationships with the original image, and even contains the parameter relationship of stitching optimization. The output accuracy can be manually adjusted according to the actual deployment requirements, and it can be used for a variety of scenarios.

[0074] Based on the same inventive concept, the embodiments of this specification also provide a panoramic image stitching device, such as Figure 4 As shown, the stitching device includes: an accuracy determination unit 401, which is used to determine the image stitching accuracy according to the hardware computing power of the target vehicle and the application scenario of the panoramic image; a data determination unit 402, which is used to obtain target mapping data from the full mapping data pre-deployed on the target vehicle according to the image stitching accuracy, and the full mapping data includes the full mapping relationship of each stitching pixel point in the panoramic image with respect to M-way surround view cameras, and the full mapping relationship of the same stitching pixel point with respect to the M-way surround view cameras is sorted according to the mapping weight, and M is an integer greater than 1; an image stitching unit 403, which is used to obtain M-way original images collected by the M-way surround view cameras installed on the target vehicle at the current moment, and stitch the M-way original images according to the target mapping data to obtain the current frame panoramic image.

[0075] In some embodiments, a pre-generation unit for pre-generating the full amount of mapping data is further included, and the pre-generation unit may include:

[0076] A determination subunit, used to determine the parameter calibration results of the M-way surround view cameras installed on the target vehicle and the area occupied by the target vehicle in the panoramic image;

[0077] An entry generation subunit, used to generate an image stitching function entry according to the parameter calibration result and the occupied area;

[0078] An image generation subunit, configured to generate M grayscale images matching the imaging size of the M-way surround view cameras according to the parameter calibration result;

[0079] The image processing subunit is used to process the M grayscale images according to the image stitching function entrance to obtain the full mapping data.

[0080] In some embodiments, the determination subunit can be used to: calibrate the camera intrinsic parameters and camera extrinsic parameters of each of the M surround view cameras to obtain the internal and external parameter calibration results of the M surround view cameras; and determine the mapping optimization parameters for the overlapping imaging area of ​​each two adjacent surround view cameras in the M surround view cameras based on the internal and external parameter calibration results.

[0081] In some embodiments, the determination subunit can be used to: determine the overlapping imaging area of ​​each two adjacent surround-view cameras among the M surround-view cameras based on the internal and external parameter calibration results to obtain M overlapping imaging areas; and determine the color weight of the pixel with respect to the two adjacent surround-view cameras for each pixel point in each overlapping imaging area of ​​the M overlapping imaging areas.

[0082] In some embodiments, the determination subunit can be used to: determine the angular offset between the line between the pixel point and the center point of the area occupied by the target vehicle in the panoramic image and the central axis of the two adjacent surround-view cameras to obtain two angular offsets of the pixel point with respect to the two adjacent surround-view cameras; normalize the two angular offsets of the pixel point with respect to the two adjacent surround-view cameras to obtain two color weights of the pixel point with respect to the two adjacent surround-view cameras.

[0083] In some embodiments, the image generation subunit can be used to: for the M surround-view cameras installed on the target vehicle, generate a grayscale image that is equal to the imaging size of the surround-view camera and inherits the internal and external calibration parameters of the surround-view camera according to the internal and external calibration results of the surround-view cameras, and the pixel value of each pixel point of the grayscale image is zero.

[0084] In some embodiments, the image processing subunit can be used to: traverse the pixels of the M grayscale images one by one through the image stitching function entrance; for each pixel traversed, modify the grayscale value of the pixel traversed currently to obtain a new grayscale image, stitch the new grayscale image with other M-1 grayscale images to obtain a current stitched image, and determine the mapping data of each target pixel in the current stitched image, and the pixel value of the target pixel is not zero; after the traversal of the M grayscale images is completed, based on the mapping data obtained by modifying a pixel in the M grayscale images each time, combine to obtain the full mapping relationship of each stitched pixel; for each stitched pixel, sort the full mapping relationships belonging to the stitched pixel from large to small according to the mapping weight to obtain the full mapping data.

[0085] In some embodiments, the full mapping relationship of each stitched pixel point includes: in the original image exposed by M-way surround view cameras, multi-dimensional information of each original pixel point will be mapped to the stitched pixel point, and the multi-dimensional information of each original pixel point includes the coordinate information of the original pixel point, the source camera ID of the surround view camera providing the original pixel point, and the mapping weight of the original pixel point mapped to the stitched pixel point.

[0086] Based on the same inventive concept, the embodiments of this specification also provide an electronic device. Figure 5 As shown, the electronic device includes a memory 504, a processor 502, and a computer program stored in the memory 504 and executable on the processor 502. When the processor 502 executes the computer program, the panoramic image stitching method described in any of the above embodiments is implemented.

[0087] Among them, Figure 5 In the embodiment of the present invention, a bus architecture (represented by bus 500) is shown, which may include any number of interconnected buses and bridges, and bus 500 connects various circuits including one or more processors represented by processor 502 and memory represented by memory 504. Bus 500 may also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 505 provides an interface between bus 500 and receiver 501 and transmitter 503. Receiver 501 and transmitter 503 may be the same element, namely a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 502 is responsible for managing bus 500 and general processing, while memory 504 may be used to store data used by processor 502 when performing operations.

[0088] Based on the same inventive concept, an embodiment of the present specification further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned panoramic image stitching method is implemented.

[0089] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that has the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0092] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0093] Obviously, those skilled in the art can make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if these modifications and variations of this specification fall within the scope of the claims of this specification and their equivalents, this specification is also intended to include these modifications and variations.

Claims

1. A panoramic image stitching method, characterized in that: include: Determine the image stitching accuracy based on the hardware computing power of the target vehicle and the application scenario of the panoramic image; According to the image stitching accuracy, target mapping data is obtained from full mapping data pre-deployed on the target vehicle, wherein the full mapping data includes a full mapping relationship of each stitching pixel point in the panoramic image with respect to M surround view cameras, and the full mapping relationship of the same stitching pixel point with respect to the M surround view cameras is sorted according to mapping weights, where M is an integer greater than 1; M original images collected by M-way surround view cameras installed on the target vehicle at the current moment are acquired, and the M original images are spliced ​​according to the target mapping data to obtain a current frame panoramic image.

2. The method according to claim 1, characterized in that The method further comprises the step of pre-generating the full amount of mapping data: Determine parameter calibration results of M-way surround view cameras installed on the target vehicle and the area occupied by the target vehicle in the panoramic image; Generate an image stitching function entry according to the parameter calibration result and the occupied area; Generate M grayscale images matching the imaging size of the M-way surround view cameras according to the parameter calibration results; The M grayscale images are processed according to the image stitching function entrance to obtain the full mapping data.

3. The method according to claim 2, characterized in that The step of determining the calibration result of parameters of the M-way surround view cameras installed on the target vehicle includes: Calibrate the camera internal parameters and camera external parameters of each of the M surround view cameras to obtain the internal and external parameter calibration results of the M surround view cameras; According to the internal and external parameter calibration results, mapping optimization parameters for the overlapping imaging area of ​​each two adjacent surround view cameras in the M surround view cameras are determined.

4. The method according to claim 3, characterized in that Determining, based on the internal and external parameter calibration results, mapping optimization parameters for the overlapping imaging areas of each of two adjacent surround view cameras in the M surround view cameras includes: According to the internal and external parameter calibration results, determining the overlapping imaging areas of every two adjacent surround view cameras in the M surround view cameras to obtain M overlapping imaging areas; For each pixel point in each overlapping imaging area of ​​the M overlapping imaging areas, a color weight of the pixel point with respect to two adjacent surround view cameras is determined.

5. The method according to claim 4, characterized in that The determining of the color weight of the pixel point with respect to two adjacent surround view cameras includes: Determine the angular offset between the line connecting the pixel point and the center point of the area occupied by the target vehicle in the panoramic image and the central axis of the two adjacent surround view cameras, so as to obtain two angular offsets of the pixel point with respect to the two adjacent surround view cameras; The two angular offsets of the pixel point with respect to the two adjacent surround view cameras are normalized to obtain the two color weights of the pixel point with respect to the two adjacent surround view cameras.

6. The method according to claim 3, characterized in that The generating, according to the parameter calibration result, M grayscale images matching the imaging size of the M-way surround view camera comprises: For the M surround view cameras installed on the target vehicle, a grayscale image having an imaging size equal to that of the surround view camera and inheriting the internal and external calibration parameters of the surround view camera is generated according to the internal and external calibration results of the surround view cameras, and the pixel value of each pixel point of the grayscale image is zero.

7. The method according to claim 2, characterized in that The processing of the M grayscale images according to the image stitching function entry to obtain the full mapping data includes: Traversing the pixels of the M grayscale images one by one through the image stitching function entrance; For each pixel point traversed each time, modify the grayscale value of the pixel point traversed currently to obtain a new grayscale image, splice the new grayscale image with other M-1 grayscale images to obtain a current spliced ​​image, determine the mapping data of each target pixel point in the current spliced ​​image, and the pixel value of the target pixel point is not zero; After the traversal of the M grayscale images is completed, based on the mapping data obtained by modifying one pixel in the M grayscale images each time, a full mapping relationship of each spliced ​​pixel is obtained by combining; For each spliced ​​pixel point, the full amount mapping relationships belonging to the spliced ​​pixel point are sorted from large to small according to the mapping weights to obtain the full amount mapping data.

8. The method according to claim 7, characterized in that The full mapping relationship of each stitched pixel point includes: the multi-dimensional information of each original pixel point mapped to the stitched pixel point in the original image exposed by the M-way surround view camera, and the multi-dimensional information of each original pixel point includes the coordinate information of the original pixel point, the source camera ID of the surround view camera that provides the original pixel point, and the mapping weight of the original pixel point mapped to the stitched pixel point.

9. A panoramic image stitching device, characterized in that: include: An accuracy determination unit, used to determine the image stitching accuracy according to the hardware computing power of the target vehicle and the application scenario of the panoramic image; a data determination unit, configured to obtain target mapping data from full mapping data pre-deployed on the target vehicle according to the image stitching accuracy, wherein the full mapping data includes a full mapping relationship of each stitching pixel point in the panoramic image with respect to M surround view cameras, and the full mapping relationship of the same stitching pixel point with respect to the M surround view cameras is sorted according to a mapping weight, where M is an integer greater than 1; The image stitching unit is used to obtain M-way original images collected by M-way surround view cameras installed on the target vehicle at the current moment, and stitch the M-way original images according to the target mapping data to obtain a current frame panoramic image.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for stitching panoramic images described in any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the panoramic image stitching method described in any one of claims 1 to 8 is implemented.

Images