Image stitching method and device, medium, and product

Abstract

An image stitching method and device, a medium, and a product. The method comprises: determining an average pixel value of target areas in a plurality of original images, respectively, wherein each target area is an area corresponding to a stitching area in a finally generated target bird's-eye view in an original image, and the stitching area is a complete or partial overlapping area between top views in the target bird's-eye view (S101); on the basis of the average pixel value of the target areas and a preset constraint equation set, determining a compensation coefficient corresponding to each original image (S102); and performing compensation processing and stitching conversion processing on each original image on the basis of the compensation coefficient to generate a target bird's-eye view (S103). By means of the compensation coefficient generated on the basis of the average pixel value of the target areas, a color difference between the original images can be adjusted more effectively, so that a small color difference between the finally generated target bird's-eye views is achieved, thereby improving the user experience.

Description

Image stitching method, device, medium and product

[0001] The present application claims priority to the Chinese patent application No. 202410627784.4, filed on May 20, 2024, and entitled "Image stitching method, device, medium and product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and in particular to an image stitching method, device, medium and product. BACKGROUND

[0003] With the increasing intelligence of the automotive industry and the decreasing cost of vehicle-mounted cameras, multiple dispersed images can be stitched to generate an overall bird's-eye view of the virtual view angle of the roof by mounting multiple fisheye cameras. Through the bird's-eye view, the driver can more clearly identify the obstacles around the vehicle, providing convenience for driving, parking, etc.

[0004] Since different fisheye cameras are generally installed around the vehicle body, the light intensity of different cameras often differs, especially in dark, partially blocked and fast-moving scenes, the light difference is more obvious, so the original images taken by different fisheye cameras will have obvious brightness and color differences.

[0005] In such a scenario, the bird's-eye view generated by stitching the original images taken by different fisheye cameras will also present obvious color differences, resulting in poor user experience. SUMMARY

[0006] The present application provides an image stitching method, device, medium and product to solve the problem that the bird's-eye view generated by stitching the original images taken by different fisheye cameras will present obvious color differences, resulting in poor user experience.

[0007] The first aspect of the present application provides an image stitching method, comprising:

[0008] determining the average pixel value of a target region in each of the plurality of original images; the target region is a region in the original image corresponding to a stitching region in the finally generated target bird's-eye view; the stitching region is all or part of the overlapping region between each overhead image in the target bird's-eye view;

[0009] determining a compensation coefficient corresponding to each original image according to the average pixel value of each target region and a preset constraint equation set;

[0010] compensating and stitching converting each original image according to the compensation coefficient to generate a target bird's-eye view.

[0011] Further, the method as described above, the determining the average pixel value of the target region in each of the plurality of original images comprises:

[0012] converting the pixels of each of the target regions to an RGB color space; the pixels of each of the target regions being determined based on a preset mapping table and the pixels of the stitching region in the target bird's eye view;

[0013] calculating the average pixel value corresponding to each of the RGB color channels in the target region.

[0014] Further, the method as described above, the determining the average pixel value of the target region in each of the plurality of original images comprises:

[0015] for each of the target regions, the following processing is performed:

[0016] inputting the average pixel value corresponding to each of the color channels into the preset constraint equation set to construct a target constraint equation set;

[0017] solving the target constraint equation set to generate the compensation coefficient under the corresponding color channel in each of the original images; the absolute value of the compensation coefficient is less than a preset compensation threshold.

[0018] Further, the method as described above, the solving the target constraint equation set to generate the compensation coefficient under the corresponding color channel in each of the original images comprises:

[0019] if the absolute value of the function value of the target constraint equation set is greater than a preset color difference threshold, setting the compensation coefficient under the corresponding color channel in the largest original image to a preset value; the largest original image is the original image to which the target region with the largest average pixel value belongs;

[0020] solving the target constraint equation set to generate the compensation coefficient under the corresponding color channel in the other original images; the other original images are the original images other than the largest original image;

[0021] if the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, solving the target constraint equation set to generate the compensation coefficient under the corresponding color channel in each of the original images.

[0022] Further, in the target bird's eye view, the stitching region is a region formed by a new boundary line and a boundary line of the target bird's eye view after the intersection line in the target bird's eye view is rotated by a preset center angle based on a center; the new boundary line is the intersection line after being rotated by the preset center angle.

[0023] the compensation processing and the stitching conversion processing on each of the original images based on the compensation coefficient to generate the target bird's eye view comprises:

[0024] compensate the target region according to the first included angle, the preset central angle of the circle and the compensation coefficient, to generate a compensated target region; the first included angle is an angle between a line formed by each pixel in the splicing region and the center of the circle and the new boundary line;

[0025] compensate the remaining region in each original image according to the compensation coefficient, to generate a compensated remaining region; the remaining region is a region in the original image other than the target region;

[0026] splicing and converting the compensated target region and the compensated remaining region, to generate a target bird's eye view.

[0027] Further, the method described above, the compensation of the target region according to the first included angle, the preset central angle of the circle and the compensation coefficient, to generate a compensated target region, comprising:

[0028] determining the splicing weight of each pixel in the target region according to the first included angle and the preset central angle of the circle;

[0029] for each pixel in the target region, performing weighted processing on the splicing weight, the compensation coefficient and the pixel value of the corresponding pixel in the target region, to generate the pixel value of the corresponding pixel in the compensated target region;

[0030] generating the compensated target region based on the pixel value of each pixel in the compensated target region.

[0031] Further, the method described above, the new boundary line has a corresponding relationship with the original image;

[0032] the determination of the splicing weight of each pixel in the target region according to the first included angle and the preset central angle of the circle, comprising:

[0033] determining a second included angle from the first included angle according to the corresponding relationship; the new boundary line of the second included angle corresponds to the original image to which the target region belongs;

[0034] determining the splicing weight corresponding to each pixel in the target region according to the quotient between the second included angle and twice the preset central angle of the circle.

[0035] Further, the method described above, before determining the average pixel value of the target region in each of the plurality of original images, further comprising:

[0036] obtaining the calibration parameters corresponding to the historical original image and the historical bird's eye view;

[0037] calculating the target region of the historical original image from the splicing region of the historical bird's eye view based on the calibration parameters.

[0038] The pixel mapping relationship between the splicing area and the target area is stored in a preset mapping table.

[0039] The second aspect of the present application provides an image splicing device, comprising:

[0040] A first determination module is configured to determine average pixel values of target areas in a plurality of original images respectively; the target areas are areas in the original images corresponding to a splicing area in a target bird's eye view generated finally; the splicing area is all or part of overlapping areas between the bird's eye view images in the target bird's eye view;

[0041] A second determination module is configured to determine compensation coefficients corresponding to the original images according to the average pixel values of the target areas and a preset constraint equation set;

[0042] A generation module is configured to perform compensation processing and splicing conversion processing on the original images according to the compensation coefficients, and generate the target bird's eye view.

[0043] Further, the first determination module is specifically configured to:

[0044] convert the pixels of the target areas to an RGB color space; the pixels of the target areas are determined based on a preset mapping table and pixels of the splicing area in the target bird's eye view; and calculate average pixel values corresponding to RGB color channels in the target areas.

[0045] Further, the second determination module is specifically configured to:

[0046] For each target area, the following processing is performed:

[0047] input the average pixel values corresponding to the color channels into a corresponding preset constraint equation set to construct a target constraint equation set; and solve the target constraint equation set to generate compensation coefficients in the corresponding color channels in the original images; the absolute value of the compensation coefficients is less than a preset compensation threshold.

[0048] Further, when solving the target constraint equation set to generate the compensation coefficients in the corresponding color channels in the original images, the second determination module is specifically configured to:

[0049] if the absolute value of the function value of the target constraint equation set is greater than the preset color difference threshold value, setting the compensation coefficient of the corresponding color channel in the maximum original image as a preset numerical value; the maximum original image is an original image to which the target region with the maximum average pixel value belongs; solving the target constraint equation set to generate the compensation coefficient of the corresponding color channel in other original images; the other original images are other original images except the maximum original image; if the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold value, solving the target constraint equation set to generate the compensation coefficient of the corresponding color channel in each original image.

[0050] Further, as described above, the device, the splicing region in the target bird's eye view is a region formed by a new boundary line and a boundary line of the target bird's eye view after the intersection line in the target bird's eye view is rotated by a preset center angle to both sides based on the center; the new boundary line is the intersection line after being rotated by the preset center angle;

[0051] The generation module is specifically configured to:

[0052] According to the first included angle, the preset center angle and the compensation coefficient, the target region is compensated and processed to generate a compensated target region; the first included angle is an angle between a connecting line formed between each pixel in the splicing region and the center and the new boundary line; according to the compensation coefficient, the remaining region in each original image is compensated and processed to generate a compensated remaining region; the remaining region is a region in the original image except the target region; the compensated target region and the compensated remaining region are spliced and converted to generate the target bird's eye view.

[0053] Further, as described above, the device, the generation module, when generating the compensated target region according to the first included angle, the preset center angle and the compensation coefficient, is specifically configured to:

[0054] According to the first included angle and the preset center angle, the splicing weight of each pixel in the target region is determined; for each pixel in the target region, the splicing weight, the compensation coefficient and the pixel value of the corresponding pixel in the target region are weighted to generate the pixel value of the corresponding pixel in the compensated target region; based on the pixel values of each pixel in the compensated target region, the compensated target region is generated.

[0055] Further, as described above, the device, the new boundary line has a corresponding relationship with the original image;

[0056] The generation module, when determining the splicing weight of each pixel in the target region according to the first included angle and the preset center angle, is specifically configured to:

[0057] determine a second included angle from the first included angle according to the correspondence relationship; a new boundary line of the second included angle corresponds to an original image to which the target region belongs; determine a stitching weight corresponding to each pixel in the target region according to a quotient between the second included angle and twice a preset central angle.

[0058] Further, the apparatus as described above, the apparatus further comprises:

[0059] The mapping module is configured to: acquire calibration parameters corresponding to the historical original image and a historical bird's eye view; inversely calculate a target region of the historical original image from a stitching region of the historical bird's eye view based on the calibration parameters; and store a pixel mapping relationship between the stitching region and the target region in a preset mapping table.

[0060] The third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0061] The memory stores computer execution instructions;

[0062] The processor executes the computer execution instructions stored in the memory to implement the image stitching method according to any one of the first aspect.

[0063] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the image stitching method according to any one of the first aspect.

[0064] The fifth aspect of the present application provides a computer program product, comprising a computer program, which is executed by a processor to implement the image stitching method according to any one of the first aspect.

[0065] The image stitching method, device, medium and product provided by the present application, the method comprises: determining the average pixel value of the target region in the plurality of original images respectively; the target region is a region in the original image corresponding to the stitching region in the finally generated target bird's eye view; the stitching region is all or part of the overlapping region between each overhead image in the target bird's eye view; determining the compensation coefficient corresponding to each original image according to the average pixel value of each target region and a preset constraint equation group; performing compensation processing and stitching conversion processing on each original image according to the compensation coefficient to generate a target bird's eye view. The image stitching method of the present application, since the target region corresponds to the stitching region of the target bird's eye view, the compensation effect of the compensation coefficient determined based on the average pixel value of the target region on the target region and other regions in the original image is better, so that the color difference between each original image can be adjusted more effectively based on the compensation coefficient, the color difference of the finally generated target bird's eye view is smaller, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0066] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0067] Fig. 1a is a schematic diagram of an image stitching process according to the present application;

[0068] Fig. 1b is a schematic diagram of a bird's eye view according to the present application;

[0069] Fig. 2a is a schematic diagram of fish-eye stitching according to the present application;

[0070] Fig. 2b is a schematic diagram of fish-eye view 1 according to the present application;

[0071] Fig. 3 is a schematic diagram of a process of image stitching according to the present application;

[0072] Fig. 4 is a schematic diagram of a process of image stitching according to the present application;

[0073] Fig. 5 is a schematic diagram of a process of image stitching according to the present application;

[0074] Fig. 6 is a schematic diagram of fish-eye stitching according to the present application;

[0075] Fig. 7 is a schematic diagram of a structure of an image stitching device according to the present application;

[0076] Fig. 8 is a schematic diagram of a structure of an electronic device according to the present application.

[0077] The specific embodiments of the present application have been shown by way of example in the above-described drawings and will be described in more detail hereafter. These drawings and detailed description are not meant to restrict the scope of the present application in any way but to explain the present application to those skilled in the art by reference to a particular embodiment. DETAILED DESCRIPTION

[0078] The exemplary embodiments will be described in detail herein below with reference to the drawings. In the following description, the same drawings reference numbers are used to denote like or similar elements. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0079] The technical solutions of the present application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0080] In order to clearly understand the technical solutions of the present application, the conception process of the technical solutions is first introduced in detail. At present, the image stitching mode is shown in FIG. 1a. In the figure, four fisheye cameras, i.e., fisheye camera 1 to fisheye camera 4, are used for exemplary illustration. The fisheye cameras 1 to 4 after brightness fine-tuning shoot to generate corresponding images, i.e., images 1 to 4. At the same time, the images 1 to 4 are processed by top view stitching through camera calibration parameters to generate a top view stitching image. The top view stitching image is combined with a preset vehicle model to generate a bird's eye view image, as shown in FIG. 1b. Fisheye 1 represents the image area corresponding to the fisheye camera 1, and similarly, the other areas also correspond to the corresponding fisheye camera.

[0081] Since different fisheye cameras are generally installed around the vehicle body, the light intensity of different cameras often has differences, especially in dark, partially blocked and fast moving scenes, the light difference is more obvious, thus the images 1 to 4 generated by the fisheye cameras 1 to 4 may have large color differences, and the finally generated bird's eye view image also has obvious color differences, resulting in poor user experience.

[0082] Therefore, in view of the problem that the bird's eye view image generated by stitching the original images based on different fisheye cameras in the prior art will present obvious color differences, resulting in poor user experience, the inventors found in research that the compensation coefficient of the original image can be determined through the target area corresponding to the bird's eye view image in the original image. Since the compensation coefficient is related to the target area, the color difference of the target area can be reduced, and the color difference between the original images can be further reduced, so that the color difference of the finally generated target bird's eye view image is smaller, and the user experience is improved.

[0083] Specifically, the image stitching and generation process is as follows:

[0084] The average pixel value of the target area in each of the plurality of original images is determined, wherein the target area is the area in the original image corresponding to the stitching area in the finally generated target bird's eye view image, and the stitching area is the entire overlapping area or part of the overlapping area between each top view image in the target bird's eye view image. The compensation coefficient corresponding to each original image is determined according to the average pixel value of each target area and a preset constraint equation set. The compensation coefficient is used for compensation processing and stitching conversion processing of each original image to generate a target bird's eye view image.

[0085] The image stitching method of the present application, since the target area corresponds to the stitching area of the target bird's eye view image, the compensation coefficient determined based on the average pixel value of the target area has better compensation effect on the target area and other areas in the original image, so that the color difference between each original image can be more effectively adjusted based on the compensation coefficient, so that the color difference of the finally generated target bird's eye view image is smaller, and the user experience is improved.

[0086] Based on the above creative findings, the inventor proposes the technical solution of the present application.

[0087] The application scenario of the image stitching method provided by the embodiment of the present application is introduced below. As shown in FIG. 2a, the figure exemplarily shows each area in the bird's eye view, which are fish eye 1 corresponding area to fish eye 4 corresponding area, and a car model. Among them, A1, B1, C1, D1 are the joint areas of the bird's eye view in the image stitching method of the embodiment, the fish eye 1 corresponding area includes A1 and B1, the fish eye 2 corresponding area includes B1 and D1, the fish eye 3 corresponding area includes C1 and D1, and the fish eye 4 corresponding area includes A1 and C1. The car model is an image model representing the vehicle in the bird's eye view. The embodiment exemplarily uses a simple image model to represent the vehicle.

[0088] Exemplarily, when image stitching is needed, the following process is performed:

[0089] ①The average pixel value of the target area in the original image photographed by fish eye 1 to fish eye 4 is determined respectively. As shown in FIG. 2b, the figure only exemplarily shows the original image corresponding to fish eye 1, and the shape and target area size of the original image are only used for convenient description and exemplarily display, and the actual shape and size may be different from that in FIG. 1. The embodiment does not limit this. The target area A2 in the original image corresponds to the joint area A1 in FIG. 2a, and the target area B2 corresponds to the joint area B1 in FIG. 2a.

[0090] ②The compensation coefficient corresponding to each original image is determined according to the average pixel value of each target area and the preset constraint equation group.

[0091] ③The compensation processing and stitching conversion processing are performed on each original image according to the compensation coefficient, and the target bird's eye view is generated.

[0092] The embodiment of the present application is introduced below in conjunction with the drawings of the specification.

[0093] FIG. 3 is a flowchart of the image stitching method provided by the present application. As shown in FIG. 3, the execution subject of the embodiment of the present application is an image stitching device, which can be integrated in an electronic device, such as a vehicle terminal. The image stitching method provided by the embodiment of the present application includes the following steps:

[0094] Step S101, the average pixel value of the target area in the plurality of original images is determined respectively. The target area is the area in the original image corresponding to the stitching area in the finally generated target bird's eye view. The stitching area is the entire overlapping area or part of the overlapping area between the bird's eye view images.

[0095] In this embodiment, the splicing area can be set according to user demand or actual application demand (camera type, vehicle size, actual environmental influence, etc.), for example, only the part with large color difference can be set as the splicing area, so that when image compensation is performed subsequently, the overall efficiency of compensation can be improved by focusing on the compensation of the key area.

[0096] The original image can be obtained from other electronic devices, such as a storage device storing the original image, or from a shooting device, such as a fisheye camera, a binocular camera, etc. The embodiment is not limited in this regard. At the same time, since the shooting device is generally distributed around the vehicle and shoots the scene around the vehicle, there can be an overlapping part (common visible area) in the area shot by each shooting device. The area in the overlapping part corresponding to the splicing area in the target bird's eye view is the target area.

[0097] The overlapping part between the original images is related to the position area and the shooting angle of the shooting device. The target area corresponding to the original image can be determined based on the camera internal and external parameters, the position area and the shooting angle of the shooting device. In addition, the target area of the original image can also be determined by inverse calculation based on the splicing area of the bird's eye view and the camera internal and external parameters of the shooting device.

[0098] Optionally, in this embodiment, before S101, the mapping relationship between the target area in the original image and the splicing area in the bird's eye view can also be determined in advance, so that in S101, the target area can be directly determined according to the mapping relationship, without the need to determine the target area by inverse calculation of the splicing area of the bird's eye view in the current process, thereby improving the efficiency of determining the target area. The specific process is as follows:

[0099] Obtain the calibration parameters corresponding to the historical original image and the historical bird's eye view.

[0100] Inverse calculation of the splicing area of the historical bird's eye view to obtain the target area of the historical original image based on the calibration parameters.

[0101] Store the pixel mapping relationship between the splicing area and the target area in a preset mapping table.

[0102] In this embodiment, when the original image is converted into the bird's eye view, the image conversion is performed based on the corresponding calibration parameters. The calibration parameters include camera intrinsic parameters, camera extrinsic parameters and distortion parameters, which can be obtained by calibration methods such as linear calibration and nonlinear calibration. Thus, the target region of the historical original image can be inversely calculated from the splicing region in the historical bird's eye view through the calibration parameters. By storing the pixel mapping relationship between the splicing region and the target region in the preset mapping table, when the target region in the new original image needs to be determined in the future, the pixel mapping relationship stored in the preset mapping table can be directly determined, which improves the efficiency of determining the target region, and further improves the color optimization and image generation efficiency of the bird's eye view.

[0103] In step S102, the compensation coefficients corresponding to each original image are determined according to the average pixel value of each target region and the preset constraint equation set.

[0104] In this embodiment, the average pixel value is the average value of all pixels in a single target region. The average pixel value of each target region can be used to determine the compensation coefficient corresponding to each original image by solving the preset constraint equation set. The preset constraint equation set is an equation set with compensation coefficients and average pixel values as variables, and the function value of the preset constraint equation set satisfies a certain constraint condition. The compensation coefficient is a coefficient used to calculate the pixel value of the compensated original image when performing color compensation processing on the original image.

[0105] In order to prevent image color distortion caused by over-compensation, a threshold value of the compensation coefficient can be set to limit the compensation amount, so that the compensation coefficient obtained by solving is less than the threshold value of the compensation coefficient, thereby avoiding over-compensation of the original image. At the same time, in order to deal with the problem of excessive color difference, a color difference limit threshold value can also be set. If the function value of the preset constraint equation set is greater than the color difference limit threshold value, the compensation coefficients can be processed in a certain way, such as setting a certain compensation coefficient to a fixed value, and based on the fixed value, other compensation coefficients are solved, thereby solving the problem of excessive color difference.

[0106] In step S103, the original images are compensated and spliced and converted according to the compensation coefficients, and the target bird's eye view is generated.

[0107] In this embodiment, according to the compensation coefficients, the target region and other regions in each original image can be compensated correspondingly. For example, the pixel value of the original image can be directly multiplied by the corresponding compensation coefficient to obtain the pixel value of the compensated original image. Alternatively, the compensation coefficients can be processed according to the region type of the original image to generate optimized compensation coefficients. For example, the compensation coefficients of two original images having overlapping parts can be fused by weighting to generate fused compensation coefficients. Meanwhile, the target region can be compensated by using the fused compensation coefficients, and the specific manner can be the multiplication manner described above to obtain the pixel value of the compensated target region. For other regions, the corresponding compensation can be performed according to the compensation coefficients of the original image to generate the compensated other regions.

[0108] Meanwhile, the compensated original image is converted into a top view, and the top views are spliced to generate the corresponding target bird's eye view.

[0109] The image splicing method provided in the embodiments of the present application includes the following steps.

[0110] The image splicing method provided in the embodiments of the present application includes the following steps.

[0111] FIG. 4 is a flowchart of the image splicing method provided in the embodiments of the present application. As shown in FIG. 4, the image splicing method provided in the embodiments of the present application is based on the image splicing method provided in the previous embodiment of the present application and is further refined. Therefore, the image splicing method provided in the embodiments of the present application includes the following steps.

[0112] In step S201, the pixels of each target region are converted to the RGB color space. The pixels of the target region are determined based on a preset mapping table and the pixels of the splicing region in the target bird's eye view. The mapping relationship between the pixels of the target region and the pixels of the splicing region is recorded in the preset mapping table, and the pixels of the corresponding target region can be queried from the preset mapping table based on the pixels of the splicing region.

[0113] The video stream of the fisheye camera commonly used at present is in YUV color space, containing brightness and chrominance, and is suitable for encoding and storage. In the embodiment, the RGB color space is suitable for image acquisition and display, and by converting the pixels of the target region from other color spaces (such as YUV color space) to the RGB color space, a basis can be provided for subsequent corresponding compensation of each color channel, thereby improving the color compensation effect.

[0114] In step S202, the average pixel value corresponding to the RGB color channel in the target region is calculated.

[0115] In the embodiment, the average pixel value corresponding to each color channel in the RGB color channel is calculated, that is, the average pixel value corresponding to the R color channel, the G color channel and the B color channel exists.

[0116] In step S203, for each target region, the following processing is performed:

[0117] In step S204, the average pixel value corresponding to each color channel is input into the corresponding preset constraint equation set to construct a target constraint equation set.

[0118] In the embodiment, if the original images are four, that is, image 1, image 2, image 3 and image 4, and the corresponding vehicle regions are front, right, rear and left, respectively, the preset constraint equation set can be set as follows:

[0119] wherein G f is the function value corresponding to the G color channel of the constraint equation set, A1, B1, B2, D2, C3, D3, A4 and C4 are the average pixel values corresponding to the target regions, wherein A1 and A4 are the average pixel values of the target regions corresponding to the joint regions between image 1 and image 4, B1 and B2 are the average pixel values of the target regions corresponding to the joint regions between image 1 and image 2, C3 and C4 are the average pixel values of the target regions corresponding to the joint regions between image 3 and image 4, and D2 and D3 are the average pixel values of the target regions corresponding to the joint regions between image 2 and image 3. f1 is the compensation coefficient corresponding to image 1, f2 is the compensation coefficient corresponding to image 2, f3 is the compensation coefficient corresponding to image 3, and f4 is the compensation coefficient corresponding to image 4.

[0120] Similarly, the R channel and the B channel can also be set similarly, which will not be described here.

[0121] In step S205, the target constraint equation set is solved to generate the compensation coefficients in the corresponding color channels of each original image. The absolute value of the compensation coefficient is less than the preset compensation threshold.

[0122] In this embodiment, the solving mode can be SVD (Singular Value Decomposition), least square method, QR (QR decomposition) decomposition method, etc. The function value G is solved by any of the above modes f The optimal solution of the approximation of G = 0 is obtained, and the compensation coefficients of the corresponding color channels in each original image are obtained.

[0123] In order to avoid over-compensation and color distortion of the image, the absolute value of the compensation coefficient can be less than or equal to a preset compensation threshold. If the absolute value of the solved compensation coefficient is greater than or equal to the preset compensation threshold, the absolute value of the solved compensation coefficient can be less than or equal to the preset compensation threshold. For example, if the preset compensation threshold is 1.4 and the absolute value of the solved compensation coefficient is 1.5, the solved compensation coefficient can be equal to 1.4 or 1.39, or it can also be equal to other values less than 1.4. The preset compensation threshold can be set according to the actual camera sensor type, the size of the panorama effect, etc. For example, it can be set to 0.65 to 1.45.

[0124] Optionally, in the actual application, there are scenes with too large color difference mutations. Therefore, corresponding processing can be performed when the function value exceeds the preset color difference threshold to solve the problem of too large color difference mutations. S205 can be specifically as follows:

[0125] If the absolute value of the function value of the target constraint equation set is greater than the preset color difference threshold, the compensation coefficient of the corresponding color channel in the largest original image is set to a preset value. The largest original image is the original image to which the target region with the largest average pixel value belongs.

[0126] The target constraint equation set is solved to generate the compensation coefficient of the corresponding color channel in other original images. The other original images are other original images except the largest original image.

[0127] If the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, the target constraint equation set is solved to generate the compensation coefficient of the corresponding color channel in each original image.

[0128] In this embodiment, the preset color difference threshold can be set according to the actual camera sensor type, the size of the panorama effect, and the like, and can be set to 0.5, for example. The preset color difference threshold can be set correspondingly according to different color channels, and can be set to the same threshold or different thresholds. If the function value obtained by solving is greater than the preset color difference threshold, for example, the function value corresponding to a certain color channel is greater than the preset color difference threshold, it indicates that the color difference between the original images in this color channel is relatively large. At this time, it can be assumed that one of the original images is the true color of the ambient light, and the solution of the other original images on this basis is directly solved. In this embodiment, the original image with the largest average pixel value in this color channel, that is, the above-mentioned maximum original image, is used as the reference. Since the maximum original image is the true color of the ambient light by default, it does not need to be compensated, and therefore, the preset value corresponding to the maximum original image can be set to 1.

[0129] In step S206, the original images are compensated and stitched according to the compensation coefficient to generate a target bird's eye view.

[0130] In this embodiment, the implementation manner of S206 is similar to that of the previous embodiment S103, and will not be described here.

[0131] In order to further reduce the color difference of the joint area in the bird's eye view, in this embodiment, as shown in FIG. 5, S206 can be further refined.

[0132] In this embodiment, as shown in FIG. 6, the stitching area in the target bird's eye view is an area formed by the new boundary line and the boundary line of the target bird's eye view after the joint line in the target bird's eye view is rotated by a preset center angle to both sides based on the center, that is, the areas of A, B, C, and D in the figure.

[0133] The new boundary line is the joint line after being rotated by the preset center angle. The joint line can be set by the user in advance, for example, the diagonal line position of the bird's eye view can be set as the joint line, or it can be set inside the diagonal line, so that the image content corresponding to the fisheye camera in the left and right directions of the vehicle occupies a larger proportion, that is, the content displayed in the left and right directions of the vehicle occupies a larger proportion in the bird's eye view. The preset center angle can be set by the user in advance, for example, 20°, 30°, and the like, but generally less than 45°. The center can be the starting position of the joint line close to the vehicle body.

[0134] The image stitching method provided in this embodiment includes the following steps.

[0135] S2061, compensating the target area according to the first included angle, the preset center angle, and the compensation coefficient to generate a compensated target area. The first included angle is the angle between the connecting line between the pixels in the stitching area and the center and the new boundary line.

[0136] As shown in FIG. 6, θ represents a first included angle of a black dot (pixel) in the figure, and a is a central angle of the target region A, which is twice the preset central angle. The intersection line exemplarily shows the in the lower left region. Meanwhile, the first included angle can also be the included angle of the black dot (pixel) in the fisheye 2 direction, i.e., the included angle between the pixel in the figure and another newly added boundary line, which is a-θ.

[0137] Optionally, in the embodiment, 2061:

[0138] The stitching weight of each pixel in the target region is determined according to the first included angle and the preset central angle.

[0139] For each pixel in the target region, the stitching weight, the compensation coefficient, and the pixel value of the corresponding pixel in the target region are weighted to generate the pixel value of the corresponding pixel in the compensated target region.

[0140] The compensated target region is generated based on the pixel values of the pixels in the compensated target region.

[0141] In the embodiment, the target region exists in each original image. When generating the bird's eye view, the corresponding target regions between adjacent original images are used to generate the intersection region in the bird's eye view. As shown in FIG. 6, the image region of the fisheye 1 is generated by conversion of the original image of the fisheye 1. Similarly, the image regions of the fisheye 2 to the fisheye 4 are also generated by conversion of the corresponding original images. There is an intersection region B between the image region of the fisheye 1 and the image region of the fisheye 2, which is generated by stitching and conversion of the target region 1 of the corresponding original image of the fisheye 1 and the target region 2 of the corresponding original image of the fisheye 2.

[0142] The target region 1 and the target region 2 have corresponding stitching weights, which can be determined by the first included angle and the preset central angle. The pixel values of the pixels in the compensated target region are generated by weighted processing of the pixel values of the pixels in the target region 1 and the target region 2 based on the stitching weight and the compensation coefficient. The pixel values of the pixels in the compensated target region can be calculated based on the following algorithm:

[0143] Wherein, θ is the first included angle, a is the double central angle, rgb1 is the pixel value of the first target region, f1 is the compensation coefficient of the first target region, rgb2 is the pixel value of the second target region, and f2 is the compensation coefficient of the second target region.

[0144] Optionally, in the embodiment, the newly added boundary line has a corresponding relationship with the original image.

[0145] The process of determining the stitching weight of each pixel in the target region according to the first included angle and the preset central angle can be specifically as follows:

[0146] The second included angle is determined from the first included angle according to a correspondence relationship. The newly added boundary line of the second included angle corresponds to the original image to which the target region belongs.

[0147] A stitching weight corresponding to each pixel in the target region is determined according to a quotient between the second included angle and twice a preset central angle of the circle.

[0148] In this embodiment, each second included angle corresponds to a newly added boundary line. As shown in FIG. 6, the newly added boundary line is two lines in the joint region B, and one of the two lines is shown in the figure. The first included angle is the included angle between the line connecting the pixel and the center of the circle and the newly added boundary line, and there are two included angles. The second included angle is the included angle corresponding to the original image in the first included angle.

[0149] For example, the second included angle of fisheye 1 corresponding to the original image is θ in the figure, and the second included angle of fisheye 2 corresponding to the original image is α-θ in the figure.

[0150] According to the quotient between the second included angle and twice the preset central angle of the circle, the corresponding weight can be calculated. For example, the stitching weight of fisheye 1 corresponding to the original image is the ratio between θ and α, and the stitching weight of fisheye 2 corresponding to the original image is 1 minus the ratio.

[0151] In S2062, the remaining regions in each original image are compensated according to the compensation coefficient to generate compensated remaining regions. The remaining regions are regions in the original image other than the target region.

[0152] In this embodiment, the remaining regions do not need to be overlapped and stitched, and can be directly compensated by multiplying the pixel value by the compensation coefficient.

[0153] In S2063, the compensated target region and the compensated remaining region are stitched and converted to generate the target bird's-eye view.

[0154] The stitching and conversion processing can be that the compensated target region and the compensated remaining region are first converted in corresponding formats to generate corresponding overhead views, and then the overhead views are stitched to generate the final target bird's-eye view.

[0155] In this embodiment, under poor lighting conditions or motion scenes, the obvious stitching seam and color difference of the panoramic image caused by the traditional stitching method will cause visual interference to the driver, affecting the user experience. The image stitching method of this embodiment can automatically balance the color and brightness of multiple original images, improving the visibility and accuracy of the panoramic image. At the same time, by using different compensation schemes for the stitching area and other areas, the color difference between the stitching area and the adjacent other areas can be reduced accordingly, thereby effectively alleviating the "stitching seam" phenomenon in the stitching area of the bird's eye view and improving the user experience. For motion scenes and scenes with sudden changes in lighting, corresponding threshold values are defined to avoid overcompensation leading to color distortion and color mutation. For scenes with too large color difference, the scheme content of directly compensating other images according to the maximum original image as the reference is also added, which improves the driving experience of the driver in the whole scene.

[0156] FIG. 7 is a structural schematic diagram of an image stitching device provided by the present application. As shown in FIG. 7, the image stitching device 300 can be arranged in an electronic device in this embodiment. The image stitching device 300 comprises:

[0157] A first determination module 301 is configured to determine the average pixel value of each target region in the multiple original images respectively. The target region is a region in the original image corresponding to the stitching region in the finally generated target bird's eye view. The stitching region is the entire overlapping region or part of the overlapping region between each overhead image in the target bird's eye view.

[0158] A second determination module 302 is configured to determine the compensation coefficient corresponding to each original image according to the average pixel value of each target region and a preset constraint equation group.

[0159] A generation module 303 is configured to perform compensation processing and stitching conversion processing on each original image according to the compensation coefficient, and generate a target bird's eye view.

[0160] The image stitching device provided in this embodiment can implement the technical solutions of the method embodiment shown in FIG. 3, and has similar implementation principles and technical effects to the method embodiment shown in FIG. 3, which will not be repeated here.

[0161] The image stitching device provided in this embodiment is further refined on the basis of the image stitching device provided in the previous embodiment. The image stitching device 300 comprises:

[0162] Optionally, in this embodiment, the first determination module 301 is specifically configured to:

[0163] convert the pixels of each target region to the RGB color space. The pixels of the target region are determined based on a preset mapping table and the pixels of the stitching region in the target bird's eye view. The average pixel value corresponding to the RGB color channel in the target region is calculated.

[0164] Optionally, in the embodiment, the second determining module 302 is specifically configured to:

[0165] For each target region, the following processing is performed:

[0166] The average pixel value corresponding to each color channel is input into the corresponding preset constraint equation set to construct a target constraint equation set. The target constraint equation set is solved to generate a compensation coefficient under the corresponding color channel in each original image. The absolute value of the compensation coefficient is less than a preset compensation threshold.

[0167] Optionally, in the embodiment, when the second determining module 302 solves the target constraint equation set to generate the compensation coefficient under the corresponding color channel in each original image, the second determining module 302 is specifically configured to:

[0168] If the absolute value of the function value of the target constraint equation set is greater than a preset color difference threshold, the compensation coefficient under the corresponding color channel in the largest original image is set to a preset numerical value. The largest original image is an original image to which the target region with the largest average pixel value belongs. The target constraint equation set is solved to generate the compensation coefficient under the corresponding color channel in other original images. The other original images are other original images except the largest original image. If the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, the target constraint equation set is solved to generate the compensation coefficient under the corresponding color channel in each original image.

[0169] Optionally, in the embodiment, the splicing region in the target bird's eye view is a region formed by a new boundary line and a boundary line of the target bird's eye view after the intersection line in the target bird's eye view is rotated by a preset center angle based on the center. The new boundary line is the intersection line rotated by the preset center angle.

[0170] The generating module 303 is specifically configured to:

[0171] According to the first included angle, the preset center angle, and the compensation coefficient, the target region is compensated to generate a compensated target region. The first included angle is an angle between a connecting line formed by each pixel in the splicing region and the center and the new boundary line. According to the compensation coefficient, the remaining region in each original image is compensated to generate a compensated remaining region. The remaining region is a region in the original image except the target region. The compensated target region and the compensated remaining region are spliced and converted to generate the target bird's eye view.

[0172] Optionally, in the embodiment, when the generating module 303 compensates the target region according to the first included angle, the preset center angle, and the compensation coefficient to generate the compensated target region, the generating module 303 is specifically configured to:

[0173] The stitching weight of each pixel in the target region is determined according to the first included angle and the preset central angle of the circle. For each pixel in the target region, the stitching weight, the compensation coefficient and the pixel value of the corresponding pixel in the target region are weighted to generate the pixel value of the corresponding pixel in the compensated target region. The compensated target region is generated based on the pixel values of the pixels in the compensated target region.

[0174] Optionally, in the embodiment, the added boundary line has a corresponding relationship with the original image.

[0175] When the stitching weight of each pixel in the target region is determined according to the first included angle and the preset central angle of the circle, the generation module 303 is specifically configured to:

[0176] The second included angle is determined from the first included angle according to the corresponding relationship. The added boundary line of the second included angle corresponds to the original image to which the target region belongs. The stitching weight corresponding to each pixel in the target region is determined according to the quotient between the second included angle and twice the preset central angle of the circle.

[0177] Optionally, in the embodiment, the image stitching device 300 further includes:

[0178] The mapping module is configured to obtain the calibration parameters corresponding to the historical original image and the historical bird's eye view. The target region of the historical original image is obtained by inversely calculating the stitching region of the historical bird's eye view based on the calibration parameters. The pixel mapping relationship between the stitching region and the target region is stored in a preset mapping table.

[0179] The image stitching device provided in the embodiment can execute the technical solutions of the method embodiments shown in FIGS. 3-6, and has similar implementation principles and technical effects to the method embodiments shown in FIGS. 3-6, which will not be described again.

[0180] According to the embodiments of the present application, the present application further provides an electronic device, a computer readable storage medium and a computer program product.

[0181] As shown in FIG. 8, FIG. 8 is a structural schematic diagram of an electronic device provided by the present application. The electronic device is intended for various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, vehicle-mounted terminals, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0182] As shown in FIG. 8, the electronic device includes a processor 401 and a memory 402. The various components are connected to each other by different buses, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed within the electronic device.

[0183] The memory 402 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by the at least one processor, so that the at least one processor executes the image stitching method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the image stitching method provided by the present application.

[0184] The memory 402 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the image stitching method in the embodiment of the present application (for example, the first determining module 301, the second determining module 302 and the generating module 303 shown in FIG. 7). The processor 401 executes various function applications and data processing of the electronic device by running the non-transitory software programs, instructions and modules stored in the memory 402, that is, implements the image stitching method in the above method embodiments.

[0185] Meanwhile, the embodiment also provides a computer product. When the instructions in the computer product are executed by the processor of the electronic device, the electronic device can execute the image stitching method of the above embodiments.

[0186] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present application falling within the general scope of the application. The present application includes common knowledge or conventional technical means in the art that are not disclosed in the present application, which follow the general principles of the present application.

[0187] It should be understood that the present application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present application. The scope of the present application is only limited by the appended claims.

Claims

1. An image stitching method, characterized by, The method comprises the following steps: respectively determining average pixel values of target regions in multiple original images; the target regions are regions in the original images corresponding to splicing regions in a final generated target bird's eye view; the splicing regions are all or part of overlapping regions between each overhead image in the target bird's eye view; determining compensation coefficients corresponding to each original image according to the average pixel values of each target region and a preset constraint equation set; performing compensation processing and splicing conversion processing on each original image according to the compensation coefficients to generate the target bird's eye view.

2. The method of claim 1, wherein, The method of respectively determining average pixel values of target regions in multiple original images comprises the following steps: converting pixels of each target region to an RGB color space; the pixels of the target region are determined based on a preset mapping table and pixels of a splicing region in a target bird's eye view; calculating average pixel values corresponding to RGB color channels in the target region.

3. The method of claim 2, wherein, The method of determining compensation coefficients corresponding to each original image according to the average pixel values of each target region and a preset constraint equation set comprises the following steps: for each target region, the following processing is performed: inputting the average pixel values corresponding to each color channel into a corresponding preset constraint equation set to construct a target constraint equation set; solving the target constraint equation set to generate compensation coefficients in a corresponding color channel in each original image; the absolute value of the compensation coefficient is less than a preset compensation threshold.

4. The method of claim 3, wherein, The method of solving the target constraint equation set to generate compensation coefficients in a corresponding color channel in each original image comprises the following steps: if the absolute value of a function value of the target constraint equation set is greater than a preset color difference threshold, setting the compensation coefficient in the corresponding color channel in the largest original image to a preset value; the largest original image is an original image to which a target region with the largest average pixel value belongs; solving the target constraint equation set to generate compensation coefficients in the corresponding color channel in other original images; the other original images are original images other than the largest original image; if the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold, solving the target constraint equation set to generate compensation coefficients in the corresponding color channel in each original image.

5. The method of claim 1, wherein, The splicing region in the target bird's eye view is a region formed by a new boundary line and a boundary line of the target bird's eye view after an intersection line in the target bird's eye view is rotated by a preset center angle based on a center of the circle to both sides; The method of generating the target bird's eye view by performing compensation processing and splicing conversion processing on each original image according to the compensation coefficients comprises the following steps: performing compensation processing on the target region according to a first included angle, the preset center angle and the compensation coefficients to generate a compensated target region; the first included angle is an angle between a connecting line formed by each pixel in the splicing region and the center of the circle and the new boundary line; performing compensation processing on a remaining region in each original image according to the compensation coefficients to generate a compensated remaining region; the remaining region is a region in the original image other than the target region; performing splicing conversion processing on the compensated target region and the compensated remaining region to generate the target bird's eye view.

6. The method of claim 5, wherein, The compensation processing on the target region according to the first included angle, the preset central angle of the circle and the compensation coefficient includes: Determine the stitching weight of each pixel in the target region according to the first included angle and the preset central angle of the circle; For each pixel in the target region, the stitching weight, the compensation coefficient and the pixel value of the corresponding pixel in the target region are weighted to generate the pixel value of the corresponding pixel in the compensated target region; Generate the compensated target region based on the pixel value of each pixel in the compensated target region.

7. The method of claim 6, wherein, The stitching weight of each pixel in the target region is determined according to the first included angle and the preset central angle, including: Determine the second included angle from the first included angle according to the corresponding relationship, and the new boundary line of the second included angle corresponds to the original image to which the target region belongs; Determine the stitching weight corresponding to each pixel in the target region according to the quotient between the second included angle and twice the preset central angle.

8. The method according to any one of claims 2 to 7, characterized in that, Before determining the average pixel value of the target region in the plurality of original images respectively, further comprising: Obtain the calibration parameters corresponding to the historical original image and the historical bird's eye view; Based on the calibration parameters, the target region of the historical original image is obtained by inversely calculating the stitching region of the historical bird's eye view; Store the pixel mapping relationship between the stitching region and the target region in the preset mapping table.

9. An image stitching apparatus characterized by comprising: Including: The first determination module is configured to determine the average pixel value of the target region in the plurality of original images respectively; The target region is a region in the original image corresponding to the stitching region in the finally generated target bird's eye view; The stitching region is all or part of the overlapping region between the target bird's eye view and the target bird's eye view; The second determination module is configured to determine the compensation coefficient corresponding to each original image according to the average pixel value of each target region and the preset constraint equation set; The generation module is configured to compensate and stitch the original image according to the compensation coefficient to generate the target bird's eye view.

10. The apparatus of claim 9, wherein, The first determination module is specifically configured to: Convert the pixels of each target region to the RGB color space; the pixels of the target region are determined based on the preset mapping table and the pixels of the stitching region in the target bird's eye view; calculate the average pixel value corresponding to the RGB color channel in the target region.

11. The apparatus of claim 10, wherein, The second determination module is specifically configured to: For each target region, the following processing is performed: Input the average pixel value corresponding to each color channel into the corresponding preset constraint equation set to construct a target constraint equation set; solve the target constraint equation set to generate the compensation coefficient of each original image under the corresponding color channel; the absolute value of the compensation coefficient is less than the preset compensation threshold.

12. The apparatus of claim 11, wherein, When the second determination module solves the target constraint equation set to generate the compensation coefficient of each original image under the corresponding color channel, it is specifically configured to: If an absolute value of a function value of the target constraint equation set is greater than a preset color difference threshold value, a compensation coefficient of a corresponding color channel in a maximum original image is set as a preset numerical value; the maximum original image is an original image to which a target region with a maximum average pixel value belongs; the target constraint equation set is solved to generate compensation coefficients of the corresponding color channel in other original images; the other original images are other original images except the maximum original image; if the absolute value of the function value of the target constraint equation set is less than or equal to the preset color difference threshold value, the target constraint equation set is solved to generate compensation coefficients of the corresponding color channel in each original image.

13. The apparatus of claim 9, wherein, The splicing region in the target bird's-eye view is a region formed by a new boundary line and a boundary line of the target bird's-eye view after an intersection line in the target bird's-eye view is respectively rotated by a preset center angle to two sides, and the new boundary line is the intersection line after being rotated by the preset center angle. The generation module is specifically configured to: compensate the target region according to the first included angle, the preset center angle and the compensation coefficient to generate a compensated target region; the first included angle is an angle between a connecting line formed between each pixel in the splicing region and a center and the new boundary line; compensate remaining regions in each original image according to the compensation coefficient to generate compensated remaining regions; the remaining regions are regions in the original image except the target region; and perform splicing conversion processing on the compensated target region and the compensated remaining regions to generate the target bird's-eye view. When the generation module compensates the target region according to the first included angle, the preset center angle and the compensation coefficient to generate the compensated target region, it is specifically configured to:

14. The apparatus of claim 13, wherein, determine a splicing weight of each pixel in the target region according to the first included angle and the preset center angle; and perform weighted processing on the splicing weight, the compensation coefficient and a pixel value of a corresponding pixel in the target region to generate a pixel value of the corresponding pixel in the compensated target region. generate the compensated target region based on the pixel values of the pixels in the compensated target region. The new boundary line has a corresponding relationship with the original image.

15. The apparatus of claim 14, wherein, When the generation module determines the splicing weight of each pixel in the target region according to the first included angle and the preset center angle, it is specifically configured to: determine a second included angle from the first included angle according to the corresponding relationship; the new boundary line of the second included angle corresponds to the original image to which the target region belongs; and determine the splicing weight corresponding to each pixel in the target region according to a quotient between the second included angle and twice the preset center angle. The image splicing device further includes:

16. The apparatus of any one of claims 9-15, wherein, a mapping module configured to acquire calibration parameters corresponding to historical original images and historical bird's-eye views; inversely calculate a target region of a historical original image from a splicing region of a historical bird's-eye view based on the calibration parameters; and store a pixel mapping relationship between the splicing region and the target region in a preset mapping table. comprise:

17. An electronic device, comprising: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the image splicing method according to any one of claims 1 to 8. ​ 18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image stitching method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the image stitching method according to any one of claims 1 to 8.

Images